JLD2 MethodError when dealing with large dictionaries?

@DocDave wrote:

I have a numeric analysis code currently running on a cluster which serialises a dictionary contains several multi-million point arrays using the JLD2 library. The file is saved using @save filename source and read back using @load filename source. In previous runs using smaller arrays this has worked fine, however it has begun throwing a lengthy MethodError with these larger arrays, the first few lines of which are as follows:

ERROR: MethodError: no method matching read_array(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::JLD2.FixedLengthString{String}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1})
Closest candidates are:
  read_array(::JLD2.JLDFile, ::JLD2.ReadDataspace, ::JLD2.ReadRepresentation{T,RR}, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Union{Nothing, Array{JLD2.ReadAttribute,1}}) where {T, RR} at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:312
Stacktrace:
 [1] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.FixedLengthString{String}, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:173
 [2] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:156 [inlined]
 [3] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/datatypes.jl:76 [inlined]
 [4] read_data(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.ReadDataspace, ::UInt8, ::Int64, ::Int64, ::Int64, ::UInt16, ::JLD2.RelOffset, ::Array{JLD2.ReadAttribute,1}) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:152
 [5] load_dataset(::JLD2.JLDFile{JLD2.MmapIO}, ::JLD2.RelOffset) at /home/dave/.julia/packages/JLD2/I2BGy/src/datasets.jl:92
 [6] jlconvert at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:674 [inlined]
 [7] macro expansion at /home/dave/.julia/packages/JLD2/I2BGy/src/data.jl:1325 [inlined]

I have been struggling to resolve this issue for several days now and cannot find anything online which explains this behaviour. The current file is only 1GB but does JLD2 have a limit on file size? Is this just a limitation of the @save/@load macros? Any help would be greatly appreciated!

Version details:

$ julia -v
julia version 1.3.1

julia> Pkg.installed()
  "JLD2"             => v"0.1.11"

Posts: 1

Participants: 1

Read full topic