Hello,
I have to compare numbers that are quite small – something like 10e-5 or 10e-13 etc. Is there a way to extract the exponent (or characteristic) of the numbers, so that I will be working with -5 and -13 only? (thus passing from a float64 to a more friendly number)
Is it possible to extract such characteristic from a normal number? or instance, 2 should give 0 (as in 2x10e0).
I would I convert a number into scientific? thus 2 into 2e0?
Thank you
size of original table(101, 101, 101, 101)
MethodError: no method matching -(::NTuple{4,Int64}, ::Int64)
Closest candidates are:
-(!Matched::Complex{Bool}, ::Real) at complex.jl:299
-(!Matched::Missing, ::Number) at missing.jl:94
-(!Matched::Base.CoreLogging.LogLevel, ::Integer) at logging.jl:107
...
Stacktrace:
[1] top-level scope at In[9]:5
I’m trying to apply a matrix A to a single column of each element of X and accumulate into X, where X is a tuple of arrays.
N = 2
A = rand(N,N)
X = (ones(N,N),ones(N,N))
for fld in eachindex(X)
X[fld][:,1] += A*X[fld][:,1]
end
I can apply A*X[fld][:,1] to all fields using map and broadcasted getindex, but I get an error if I try to set or accumulate into the columns of elements of the tuple
getindex.(X,(:),1) .+= map(x->A*x,getindex.(X,(:),1)) # doesn't work
Is there a way to do this without looping over the elements of the struct?
P is an m x n matrix
Q is an m x m matrix, but only has diagonal elements, i.e.
Q[i,j] = q[k], i=j=k
Q[i,j] = 0, i != j
and eventually I have to multiply Q and P,
R = Q * P
Naturally, m in my situation is a fairly large number, ~1000, which, if Diagonal is actually stored as a full Matrix is a million elements. I really expect that it is NOT stored that way but is stored as a Diagonal type which is associated with a vector of 1000 elements, i.e. it uses 1000 memory locations.
The values of Q are going to be updated via iteration, therefore allocate once and update is the way to go. There are two ways to do this,
Create the diagonal matrix,
Q = Diagonal(q)
and then update directly,
for i=1:m
Q[i,i] = update(i)
end
R = Q * P
or update the base vector and create the Diagonal again before multiplying,
for i=1:m
q[i] = update(i)
end
R = Diagonal(q) * P
which of course seems like a bad idea since i’m creating the Diagonal instance on every iteration…
I’m really expecting the right answer to be “directly update the Diagonal matrix Q”, just wanted to be sure.
I am trying to generate a regular-directed graph and eventually a small-world network using watts_strogatz function in the Light Graphs’ package. For a network with 10,000 nodes, the time taken is as below:
Hi all,
I am trying to work through the Knet tutorial (https://github.com/denizyuret/Knet.jl/blob/master/tutorial/50.cnn.ipynb) which worked great, just until I needed/wanted to use gpu support i.e. use a KnetArray. I suppose the problem is related to my CUDA setup. The output of nvidia-smi looks good to me and I do not know what else to look at:
nvidia-smi
Fri Nov 8 13:39:57 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 418.87.00 Driver Version: 418.87.00 CUDA Version: 10.1 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX 108... Off | 00000000:03:00.0 On | N/A |
| 0% 43C P8 14W / 250W | 233MiB / 11169MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
| 0 2606 G /usr/bin/X 38MiB |
| 0 2805 G /usr/bin/gnome-shell 47MiB |
| 0 14149 C /opt/julia-1.2.0/bin/julia 135MiB |
+-----------------------------------------------------------------------------+
I am not sure whether this is relevant for this issue, but I am on a Opensuse Leap 15.0 system, and tried to install CUDA according to the official instructions.
Anyway, this line from the Knet tutorial fails with an error:
x = KnetArray(reshape([1.0:6.0...], (6,1,1,1)))
And I get the following error:
ERROR: AssertionError: Cannot allocate before CuArrays has been initialized.
Stacktrace:
[1] CuArrays.CuArray{UInt8,1,P} where P(::UndefInitializer, ::Tuple{Int64}) at /root/.julia/packages/CuArrays/BcKUl/src/memory.jl:120
[2] Type at /root/.julia/packages/CuArrays/BcKUl/src/array.jl:98 [inlined]
[3] Type at /root/.julia/packages/CuArrays/BcKUl/src/array.jl:99 [inlined]
[4] KnetPtrCu(::Int64) at /root/.julia/packages/Knet/HRYiN/src/cuarray.jl:90
[5] Type at /root/.julia/packages/Knet/HRYiN/src/kptr.jl:102 [inlined]
[6] Type at /root/.julia/packages/Knet/HRYiN/src/karray.jl:95 [inlined]
[7] KnetArray(::Array{Float64,4}) at /root/.julia/packages/Knet/HRYiN/src/karray.jl:93
[8] top-level scope at REPL[4]:1
Apparently, the problem is due to my CUDA setup?! At least I get a bunch of errors on pkg> test CuArrays:
Test Summary: | Pass Fail Error Total
CuArrays | 63 1 1124 1188
GPUArrays test suite | 25 431 456
construction | 15 7 22
constructors + similar | 1 1
comparison against Array | 15 1 16
conversion | 1 1
value constructors | 1 1
iterator constructors | 3 3
...
Batchnorm | 1 1
CUTENSOR | No tests
ForwardDiff | 10 9 19
UNARY | No tests
POW | 8 8
LITERAL_POW | 10 10
Broadcast Fix | 1 1
ERROR: LoadError: Some tests did not pass: 63 passed, 1 failed, 1124 errored, 0 broken.
in expression starting at /home/kzentis/.julia/packages/CuArrays/BcKUl/test/runtests.jl:23
ERROR: Package CuArrays errored during testing
This is just the end of the output, I can post it all if required.
It would be really great, if someone could point me where to search for the underlying cause of my problem.
I’m distributing entries of a Tuple to the ith entry of each array in a Tuple of Arrays.
using BenchmarkTools
N = 3
X = (zeros(N),zeros(N))
Y = [1,2]
function foo1!(X,Y,i)
for fld in eachindex(X)
X[fld][i] = Y[fld]
end
end
@btime foo1!($X,$Y,$1)
foo2!(x,y,i) = x[i] = y
@btime foo2!.($X,$Y,$1)
foo3!(X,Y,i) = getindex(X,i) = Y
@btime foo3!($X,$Y,$1)
Each different approach gives a different runtimes
If this is not in the correct forum, can someone move ot to the right one? Maybe Tooling? Thanks.
I don’t know much about Jupyterlab, so I am here to learn. I read up on it, but I need some to-the-point guidance and answers. I saw it in action and was impressed, as it looks better to me than VS Code and certainly better than a poor terminal. But once I learned it’s always online, I was turned off. So here are some basic questions. I hope you will properly educate me.
What is Jupyterlab?
What are its essential functions?
Can it be used fully offline (like an offline IDE)?
If online, it makes use of some online server, including extra computing power, to process things?
This is going to sound stupid - but I have a weird question. So I’m working on a dynamic programming problem just for fun, nothing novel (been solved 20yrs ago). I’ve always found the dichotomy between recursion and loops very interesting. Sometimes I think we’re missing an operator in Computer Science… Anyways,
In Julia is it possible to dynamically dispatch a series of for-next loops? I can solve the problem I want too pretty straightforwardly with recursion but my experiences with Julia and recursion have been that there’s some performance cost(usually nothing major). I imagine this could be done via a while loop right.
But imagine we needed to track information as we’re going through the loops and things, preserve order of which loop to go to next for performance, etc. The control structure is more linear than maybe recursion but, seems clunky.
Is there a more elegant way to tie this down? Is metaprogramming too far? FoldLeft over for loops? or something crazy like that.
Edit - I also do realize iterators make life a lot easier for these tasks, I’m just trying to get at the fundamentals.
Just putting this here in event anyone else has similar issues with a linux Julia install from source on an Arch Linux kernel (this may not be an issue if installing from binaries atm). This requires root access, so may not work for those on university clusters, etc. (if you search, there are some ways to use your home directory to resolve this as well).
Distro: Manjaro Linux 18.1.2
Kernel: 5.3.8-3-MANJARO
Julia: v1.2, installed using ‘pacman’
This would have occurred with Distributions and others depending on Arpack, not just StatsPlots, just where it manifested first in my case:
julia> using StatsPlots
[ Info: Precompiling StatsPlots [f3b207a7-027a-5e70-b257-86293d7955fd]
ERROR: LoadError: No deps.jl file could be found. Please try running Pkg.build("Arpack").
Currently, the build command might fail when Julia has been built from source
and the recommendation is to use the official binaries from julialang.org.
For more info see https://github.com/JuliaLinearAlgebra/Arpack.jl/issues/5.
More details to understand if this might apply to your situation and offer resolution:
First, had installed using pacman, not from Julia source binaries:
[e@xps15 ~]$ sudo pacman -Sy julia
[sudo] password for e:
:: Synchronizing package databases...
core is up to date
extra is up to date
community is up to date
multilib is up to date
resolving dependencies...
looking for conflicting packages...
:: openblas and blas are in conflict. Remove blas? [y/N] y
Packages (6) blas-3.8.0-2 [removal] cblas-3.8.0-2 http-parser-2.9.2-1 libgit2-1:0.28.3-1 openblas-0.3.7-1 julia-2:1.2.0-1.2
Total Download Size: 31.87 MiB
Total Installed Size: 203.86 MiB
Net Upgrade Size: 203.53 MiB
:: Proceed with installation? [Y/n] Y
:: Retrieving packages...
cblas-3.8.0-2-x86_64 30.1 KiB 9.78M/s 00:00
...
(5/5) installing julia [################################################################] 100%
Optional dependencies for julia
gnuplot: If using the Gaston Package from julia
:: Running post-transaction hooks...
(1/3) Updating icon theme caches...
(2/3) Arming ConditionNeedsUpdate...
(3/3) Updating the desktop file MIME type cache...
[e@xps15 ~]$ julia
_
_ _ _(_)_ | Documentation: https://docs.julialang.org
(_) | (_) (_) |
_ _ _| |_ __ _ | Type "?" for help, "]?" for Pkg help.
| | | | | | |/ _` | |
| | |_| | | | (_| | | Version 1.2.0 (2019-08-20)
_/ |\__'_|_|_|\__'_| |
|__/ |
julia>
All seemed fine, and was using Julia v1.2 for several days with other packages that didn’t require Arpack, so didn’t notice the issue at first.
Arpack seemed to work fine with adding the package, but failed here:
(v1.2) pkg> build Arpack
Building Arpack → `~/.julia/packages/Arpack/cu5By/deps/build.log`
┌ Error: Error building `Arpack`:
│ ERROR: LoadError: LibraryProduct(nothing, ["libarpack"], :libarpack, "Prefix(/home/e/.julia/packages/Arpack/cu5By/deps/usr)") is not satisfied, cannot generate deps.jl!
...
Couldn’t remove it using rm as apparently it wasn’t actually added:
(v1.2) pkg> rm Arpack
ERROR: The following package names could not be resolved:
* Arpack (7d9fca2a-8960-54d3-9f78-7d1dccf2cb97 in manifest but not in project)
Please specify by known `name=uuid`.
(v1.2) pkg> rm 7d9fca2a-8960-54d3-9f78-7d1dccf2cb97
┌ Warning: `7d9fca2a-8960-54d3-9f78-7d1dccf2cb97` not in project, ignoring
└ @ Pkg.Operations /build/julia/src/julia-1.2.0/usr/share/julia/stdlib/v1.2/Pkg/src/Operations.jl:873
[ Info: No changes
also tried to update all packages, and also to install from master and build Arpack, no change (wasn’t addressing root issue):
(v1.2) pkg> up
(v1.2) pkg> add Arpack#master
So in linux shell, reinstalled Arpack from Arch repository to be safe (it was up to date, no change in building Arpack within Julia):
~/.../cu5By/deps >>> gcc --version
gcc (GCC) 9.2.0
Copyright (C) 2019 Free Software Foundation, Inc.
This is free software; see the source for copying conditions. There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.
~/.../cu5By/deps >>> sudo pacman -S arpack
warning: arpack-3.7.0-2 is up to date -- reinstalling
I write this because other recommendations were to reinstall, ‘dev’ packages, and all sorts of other methods that would have probably made things worse. There is much discussion on this Arpack issue, and it took couple hours to read enough to understand what was going on. Hoping if you had this issue you found this note and it resolved for you quickly!
These two posts helped and offer great detail if interested:
The location where the command used to resolve was found (comment-437869878):
Wondering if there’s a simple way to run a Julia session, or use a specific package, with root privilege?
Would like to use the ODBC.jl package as sudo/root.
Am using Julia v1.2 and Linux (Arch-based platform with Manjaro 18.1.2/5.3.8-3-MANJARO x86_64)
Noted that I can run a Julia session as root, but that the packages from my regular user are no longer available:
$ sudo julia
julia> using ODBC
ERROR: ArgumentError: Package ODBC not found in current path:
- Run `import Pkg; Pkg.add("ODBC")` to install the ODBC package.
julia> exit()
Was going to give up there, but then saw this comment from @ExpandingMan in the “Discussion: Plans for Julia as a general-purpose language” thread:
So, thinking could either:
run as root, add packages needed for the query/processing. Just worried about corrupting my general environment or other unforeseen problems.
use the ]dev optioned mentioned by ExpandingMan. I just haven’t done that before.
Suggestions, or am I just asking for more trouble with any of this?
ps - I admit this is a temporary hack and not my “real ‘root’ problem”, which I believe to be improper install and configuration of my MariaDB install and config in linux, with the MariaDB data directory pointing to my home user so the mysql user can’t access, but until I can understand and deal with that, am hoping for a short-term fix, as already have a Julia workflow with a number of queries and data processing work already done from when using Win environment.
Funny, it seems a number of people have this permissions issue out in the wild apart from my case here, with other interfaces (not Julia problems), as the error message is same and scenario similar.
Trying to use the /MySQLMariaDB DSN works from a shell using root, but not in my Julia session:
Julia:
# not working yet, edit username / psswd; TODO figure out how to read this from a local file, ideally an encrypted one.
conn = ODBC.DSN("mariadb-dsn", "e", "XXXX")#update with your username and password. Can then edit and remove password after code block has run.
errors with:
[ODBC] 08S01: [unixODBC][MySQL][ODBC 8.0(w) Driver]Can't connect to local MySQL server through socket '/tmp/mysql.sock' (13)
ODBC.ODBCError("ODBC.API.SQLConnect(dbc, connectionstring, username, password) failed; return code: -1 => SQL_ERROR")
Stacktrace:
[1] #DSN#3(::Bool, ::Type{ODBC.DSN}, ::String, ::String, ::String) at /home/e/.julia/packages/ODBC/YEzHX/src/ODBC.jl:57
[2] ODBC.DSN(::String, ::String, ::String) at /home/e/.julia/packages/ODBC/YEzHX/src/ODBC.jl:119
[3] top-level scope at In[7]:1
And using a local MySQL command line client (tried a couple, I like mycli as it offers syntax highlighting and auto-suggest in a shell environment!). These clients:
BUT, running isql without root privilege give me the same error as within Julia the ODBC, hence, why I just want to get by for now running as root.
~ >>> isql -v mariadb-dsn
[08S01][unixODBC][MySQL][ODBC 8.0(w) Driver]Can't connect to local MySQL server through socket '/tmp/mysql.sock' (13)
[ISQL]ERROR: Could not SQLConnect
For some reason @debug messages are appearing when I run a modules function via shift+enter in Juno with Julia 1.2 but not when I run the function directly on the REPL.
Here’s the minimal setup:
Create new package via generate LoggerProblem at the julia repl.
Open the new folder LoggerProblem in Juno.
Change src/LoggerProblem.jl to contain the following:
module LoggerProblem
export log_example
function log_example()
@debug "This statement should appear at debug level."
@warn "This statement should appear at warn level."
end
end # module
Create a new file scratch.jl in the package directory with the following contents:
import Pkg
Pkg.activate(".")
using Revise
using LoggerProblem
using Logging
logger = ConsoleLogger(stdout, Logging.Debug)
old_logger = global_logger(logger)
log_example()
If you run each line using shift+enter you get the following output.
┌ Debug: This statement should appear at debug level.
└ @ LoggerProblem C:\Users\carlm\workspace\julia-lisp\LoggerProblem\LoggerProblem\src\LoggerProblem.jl:6
┌ Warning: This statement should appear at warn level.
└ @ LoggerProblem C:\Users\carlm\workspace\julia-lisp\LoggerProblem\LoggerProblem\src\LoggerProblem.jl:7
However, if I run the next line I get different results:
julia> log_example()
┌ Warning: This statement should appear at warn level.
└ @ LoggerProblem C:\Users\carlm\workspace\julia-lisp\LoggerProblem\LoggerProblem\src\LoggerProblem.jl:7
I’ve read several posts and the documentation, but I can’t tell what’s going wrong. Any suggestions on where to look next for troubleshooting?
As can be seen, the last known value of y is stored in all of m instead of the different values of y in each iteration. Is there something wrong in my understanding?
I am trying to understand inplace assignment in julia.
In the following example I want to update a and b. The following code works, and it updates a and b in place without any allocations.
n = 1000
a = ones(n,1)
b = ones(n,1)
c = 1.0
d = 2.0
nsel = rand(n) .> 0.5
function testalloc!(a, b, c, d, nsel)
@. a = c * a + d * b
@. b = d * a + c * b
end
@btime testalloc!(a, b, c, d, nsel)
891.646 ns (0 allocations: 0 bytes)
However, if i have to do it for a subset there are 28 allocations and it is slower
function testalloc1!(a, b, c, d, nsel)
@. a[nsel] = c * a[nsel] + d * b[nsel]
@. b[nsel] = d * a[nsel] + c * b[nsel]
end
@btime testalloc1!(a, b, c, d, nsel)
3.350 μs (28 allocations: 24.22 KiB)
and if I wanted to do for the complementary subset there are 40 allocations… and it is even more slower
function testalloc2!(a, b, c, d, nsel)
@. a[.!nsel] = c * a[.!nsel] + d * b[.!nsel]
@. b[.!nsel] = d * a[.!nsel] + c * b[.!nsel]
end
@btime testalloc2!(a, b, c, d, nsel)
3.775 μs (40 allocations: 27.13 KiB)
Could someone please help me understand the correct way to update subsets of arrays.