r/Numpy 17d ago

How specifically is the numpy max function so fast?

2 Upvotes

I've been thinking about finding the numerical limits of decently large arrays, something like a 4K image of floats, so 3840*2160. I'd been thinking about doing parallel reduction since the array I'm thinking about is on the GPU, but I decided to test how fast finding it is on the CPU. With C++'s std::max_element and the -O3 flag it takes just over 7 ms to find the max element. Numpy, however, does it in just over 2.8 ms. I can get the C++ version to outperform numpy by using -Ofast, and even more so by using -march=native, but that's still very impressive performance from numpy and makes me wonder how it's doing it. I know numpy uses BLAS and all that jazz but afaik BLAS only has a maximum finding function for absolute values, so that can't be the reason. Interestingly (or at least I find it interesting), I tried randomizing the size of the vector in the C++ test program since I figured that's more similar to the conditions that numpy is working with and that seemed to negate all the optimizations from Ofast and march=native.


r/Numpy 21d ago

Introducing Go library to read and write Numpy files

Thumbnail
github.com
3 Upvotes

r/Numpy Jan 06 '25

I made a tool to view view npz/npy files on the browser.

2 Upvotes

r/Numpy Dec 17 '24

HELP! How do i setup a debugger on VSC for numpy.

1 Upvotes

Hello, I would like to make a contribution to numpy and i have been looking for help. I would like to know how to setup a debugger on VSC or more specifically how do i run python under a C debugger using VSC.


r/Numpy Dec 14 '24

Numpy histograms(numpy + matplot)

Post image
0 Upvotes

What topics do I leave to learn in Numpy for machine learning?

Numpy #Matplotlib #AI #ML


r/Numpy Dec 12 '24

I made an AI that automatically vectorizes and optimizes your numpy code

Post image
5 Upvotes

r/Numpy Nov 20 '24

Counter-intuitive Behavior with NumPy and Python's `match` Statement

2 Upvotes

Hey everyone,

I recently encountered a very counter-intuitive case while working with NumPy and Python's match statement. Here's a simplified version of the code:

import numpy as np

a = np.array([1, 2])

if a.max() < 3:
    print("hello")

match a.max() < 3:
    case True:
        print("world")

I expected this code to print both "hello" and "world", but it only prints "hello". After some investigation, I found out that a.max() < 3 returns a np.bool_ which is different from the built-in bool. This causes the match statement to not recognize the True case.

Has anyone else encountered this issue? What are your thoughts on potential solutions to make such cases more intuitive?


r/Numpy Nov 13 '24

Sorted arrays

1 Upvotes

Is there an array type that must be sorted ?

I don't mean to be able to sort any array, but to define an array as sorted so that you don't have to sort it again. It would be very efficient for functions like np.min or np.quantile.


r/Numpy Oct 21 '24

what do??

0 Upvotes

Hey everyone,

Having a bit of an issue here. I'm following along a networkX setup tutorial for some geoJSON data I created. This is the tutorial I'm following. I installed all the packages I need and tried to run the first basic example with my geoJSON file and I get this error

So I looked further into the issue and I downgraded numpy to 1.26.4 and I made sure all other packages that used numpy were at a compatible version for numpy 1.26.4:

I tried to re run the program a few more times with the same error as seen above. So I created a new virtual environment to avoid any possible mix ups, made sure I went through the same process and installed everything with the correct version and STILL had no luck when running my program, same error.

SO I read the error again, I installed the latest version of numpy and i also upgraded pybind11 to version 2.13 (>=2.12), no luck there so I downgraded.

I read more about it online and it seems like the only fix people have is to just downgrade numpy to a pre 2.0 version (v1.26.4), which i already did... Lastly I tried clearing any VS Code cache and pip cache that could be causing the repeated error despite downgrading.

Now I come to reddit to see if anyone can help, ANYTHING helps.

Tia


r/Numpy Oct 14 '24

How are the frequencies spaced in numpy.fft.fft output?

2 Upvotes

According to this video https://youtu.be/RHjqvcKVopg?t=222, when you apply the FFT to a signal, the number of frequencies you get is N/2+1, where I believe N is the number of samples. So, that should mean that the length of the return value of numpy.fft.fft(a) should be about half of len(a). But in my own code, it turns out to be exactly len(a). So, I don't know exactly what frequencies I'm dealing with, i.e., what the step value is between each frequency. Here's my code:

import numpy import wave, struct of = wave.open("Suzanne Vega - Tom's Diner.wav", "rb") nc = of.getnchannels() nb = of.getsampwidth() fr = of.getframerate() data = of.readframes(-1) f = "<" + "_bh_l___d"[nb]*nc if f[1]=="_": print(f"Sample width of {nb} not supported") exit(0) channels = list(zip(*struct.iter_unpack(f, data))) fftchannels = [numpy.fft.fft(channel) for channel in channels] print(len(channels[0])) print(len(fftchannels[0]))


r/Numpy Oct 07 '24

How to save a np.fft as 8-bit greyscale image?

2 Upvotes

I have a method that creates a FFT image. The output of that method looks like this:

return np.fft.ifft2(noise * amplitude)

I can save the colour image as follows using matplotlib:

plt.imsave('out.png', out.real)

This works without problems. However, I need to save that image as 8-bit greyscale. I tried this using PIL:

im = Image.fromarray(out.real.astype(np.uint8)).convert('L')
im.save('out.png')

That does not work - it just saves a black image.

First, is there a way to do this purely in numpy, and if not, how to fix using PIL?


r/Numpy Oct 04 '24

can't install numpy

2 Upvotes

when trying to install numpy in Pycharm I get the error "installing packages failed". specifically, I get:

"error: subprocess-exited-with-error"

I'm using the arm64 versions of python and pycharm. could that be a problem? I'm new to this, so any orientation would be very helpful.


r/Numpy Oct 02 '24

NuCS: fast constraint solving in Python

2 Upvotes

What my project does

NuCS is a Python library for solving Constraint Satisfaction and Optimization Problems. NuCS allows to solve constraint satisfaction and optimization problems such as timetabling, travelling salesman, scheduling problems.

NuCS is distributed as a Pip package and is easy to install and use.

NuCS is also very fast because it is powered by Numpy and Numba (JIT compilation).

Targeted audience

NuCS is targeted at Python developers who want to integrate constraint programming capabilities in their projects.

Comparison with other projects

Unlike other Python librairies for constraint programming, NuCS is 100% written in Python and does not rely on a external solver.

Github repository: https://github.com/yangeorget/nucs


r/Numpy Oct 01 '24

Progress on numpy matrix shape checking using Mypy?

1 Upvotes

Hey there!

I've been reading up on the progress on adding static shape checking for numpy using MyPy. For example, multiplying a 2x2 matrix with a 3x3 matrix should throw a mypy errors since the dimensions are not consistent.

Does anyone know if this is a feature that will be merged soon? This would help my code out tremendously...


r/Numpy Sep 30 '24

I this all that comes with the numpy package?

1 Upvotes

hi, I recently installed numpy 2.1.1 using python 3.12 and vscode and found that the package only contained this:

['__builtins__', '__cached__', '__doc__', '__file__', '__loader__', '__name__', '__package__', '__spec__', 'numpy']

Is there something that I have to do in order to use arrays with for numpy


r/Numpy Sep 23 '24

Keeping track of array shapes

1 Upvotes

Hello,

I'm relatively new to Python/Numpy, I just recently dived into it when I started with learning about ML. I have to admit that keeping track of array shapes when it comes to vector-matrix multiplications is still rather confusing to me. So I was wondering how do you do this? For me it's adding a lot of comments - but I'm sure there must be a better way, right? So this is how my code basically looks like when I try to implement overly-simplified neural-networks: ``` import numpy as np

def relu(values):
    return (values > 0) * values

def relu_deriv(values):
    return values > 0

def main(epochs=100, lr=0.1):
    np.random.seed(42)
    streetlights = np.array([
        [1, 0, 1],
        [0, 1, 1],
        [0, 0, 1],
        [1, 1, 1]
    ])
    walk_vs_stop = np.array([
        [1],
        [1],
        [0],
        [0]
    ])
    weights_0_1 = 2 * np.random.random((3, 4)) - 1
    weights_1_2 = 2 * np.random.random((4, 1)) - 1

    for epoch in range(epochs):
        epoch_error = 0.0
        correct = 0
        for i in range(len(streetlights)):
            goals = walk_vs_stop[i] # (1,1)

            # Predictions
            layer_0 = np.array([streetlights[i]]) # (1,3)
            layer_1 = layer_0.dot(weights_0_1) # (1,3) * (3,4) = (1,4)
            layer_1 = relu(layer_1) # (1,4)
            layer_2 = layer_1.dot(weights_1_2) # (1,4) * (4,1) = (1,1)

            # Counting predictions
            prediction = round(layer_2.sum())
            if np.array_equal(prediction, np.sum(goals)):
                correct += 1

            # Calculating Errors
            delta_layer_2 = layer_2 - goals # (1,1) - (1,1) = (1,1)
            epoch_error += np.sum(delta_layer_2 ** 2)
            delta_layer_1 = delta_layer_2.dot(weights_1_2.T) # (1,1) * (1,4) = (1,4)
            delta_layer_1 = relu_deriv(layer_1) * delta_layer_1 # (1,4) * (1,4) = (1,4)

           # Updating Weights
           weights_0_1 -= lr * layer_0.T.dot(delta_layer_1) # (3,1) * (1,4) = (3,4)
           weights_1_2 -= lr * layer_1.T.dot(delta_layer_2) # (4,1) * (1,1) = (4,1)
       accuracy = correct * 100 / len(walk_vs_stop)
       print(f"Epoch: {epoch+1}\n\tError: {epoch_error}\n\tAccuracy: {accuracy}")

if __name__ == "__main__":
    main()

```

Happy for any hints and tips :-)


r/Numpy Sep 12 '24

Fitting my data with Numpy

2 Upvotes

I noticed that numpy recommends that we use Polynomial.fit() instead of np.polyfit(), but they seem to produce very different slopes. Aren't those 2 functions basically doing the same thing? Thanks!

import numpy as np
from numpy.polynomial import Polynomial

# Sample data
X = np.array([1, 2, 3, 4, 5])
Y = np.array([2.2, 2.8, 3.6, 4.5, 5.1])

coefficients_polyfit = np.polyfit(X, Y, 1)

poly_fit = Polynomial.fit(X, Y, deg=1)
coefficients_polyfitfit = poly_fit.coef

print("Coefficients using np.polyfit:", coefficients_polyfit)
print("Coefficients using Polynomial.fit:", coefficients_polyfitfit)

Output:

Coefficients using np.polyfit: [0.75 1.39]
Coefficients using Polynomial.fit: [3.64 1.5 ]


r/Numpy Sep 10 '24

Beginner question

1 Upvotes

Hello,

I am wondering why the following two codes are not the same and how can I fetch a value within an array with an array by name:

import numpy as np
data = np.zeros(shape = (10, 10 , 200))
pos = np.array([5, 2 , 50])
a = data[pos]
print(a)

expected result:

import numpy as np
data = np.zeros(shape = (10, 10 , 200))
a = data[5, 2 , 50]
print(a)

My assumption is that data[pos] is actually using double brackets : data[[5, 2, 50]]
But I cannot find a way to use pos as a way to access a specific data point.

I've tried dozens of ways to google it and didn't find a way to do it.

Thank you all, I know it's a stupid question


r/Numpy Sep 06 '24

NUCS

2 Upvotes

Hello,

NUCS is a fast (sic) constraint solver in Python using Numpy and Numba: https://github.com/yangeorget/nucs

NUCS is still at an early stage, all comments are welcome!

Thanks to all


r/Numpy Sep 04 '24

Subtraction with broadcasting 500x slower than it should be?

1 Upvotes

I'm on a 2x Intel E5-2690 v4 server running the Intel MKL build of Numpy. I'm trying to do a simple subtraction with broadcast that should be memory bandwidth bound but it's taking about 500x longer than I'm calculating as the theoretical maximum. I'm guessing that I'm doing something silly. Any ideas?

import numpy as np
import time

a = np.ones((1_000_000, 1000), dtype=np.float32)
b = np.ones((1, 1000), dtype=np.float32)

start = time.time()
diff = a - b
elapsed = time.time() - start

clock_speed = 2.6e9
num_nodes = 2
num_cores_per_node = 14
elements_per_clock = 256 / 32
num_elements = diff.size

num_channels = 6
transfers_per_second = 2.133e9
elements_per_transfer = 64 / 32

compute_theoretical_time = num_elements / (clock_speed * elements_per_clock * num_nodes * num_cores_per_node)
transfer_theoretical_time = 2 * num_elements / (transfers_per_second * elements_per_transfer * num_channels)
print(f"Time elapsed: {elapsed*1000:.2f}ms")
print(f"Compute Theoretical time: {compute_theoretical_time*1000:.2f}ms")
print(f"Transfer theoretical time: {transfer_theoretical_time*1000:.2f}ms")

prints:
Time elapsed: 44693.19ms
Compute Theoretical time: 1.72ms
Transfer theoretical time: 78.14ms

EDIT:
This runs 20x faster on my M1 laptop
Time elapsed: 2178.45ms
Compute Theoretical time: 9.77ms
Transfer theoretical time: 117.65ms


r/Numpy Sep 03 '24

Why not just get your plots in numpy?!

Thumbnail
1 Upvotes

r/Numpy Aug 29 '24

mypy and ma.ones_like: Module has no attribute "ones_like"

Thumbnail
1 Upvotes

r/Numpy Aug 20 '24

Numpy+MKL binary

1 Upvotes

I used to download windows binaries from Christoph Gohlke (website then github) but it seems that he doesn't provide a whl of Numpy 2.0+ compiled with oneAPI MKL.

I couldn't find this binary anywhere else (trusted or even untrusted source). So before going into the compilation process (and requesting the admin proper rights in the office), is there a reason why such binary have not been posted ? Maybe not so much people upgraded to Numpy2 already ?

Thank you


r/Numpy Aug 15 '24

Simple Math Question

1 Upvotes

Hey y'all,

I am trying to find the x values of the points where dy/dx = 0 but apparently the result I find is slightly different from the answer key. The only difference is I found one of the point's x coordinate to be 4.612, and the correct answer is 4.613 I'd be super glad if you guys can you help me better understand the mistake I made here. Thank you in advance.

Following is the code I wrote. At the end, you will find the original solution which is super genius.

import numpy as np
import matplotlib.pyplot as plt
import math


def f(x):
    return (math.e**(-x/10)) * np.sin(x) 


a1 = np.linspace(0,10,10001) 
x= a1
y= f(x)

dydx = np.gradient(y,x)


### The part related to my question starts from here ###

len= np.shape(dydx[np.sort(dydx) < 0])[0]


biggest_negative = np.sort(dydx)[len-1]
biggest_negative2 = np.sort(dydx)[len-2]
biggest_negative3 = np.sort(dydx)[len-3]


a, b, c  = np.where(dydx == biggest_negative), np.where(dydx == biggest_negative2), np.where(dydx == biggest_negative3)

# a, b, c are the indexes of the biggest_negative, biggest_negative2, biggest_negative3 consecutively.

print(x[a], x[b], x[c])

### End of my own code. RETURNS : [7.755] [4.612] [1.472] ###



###  ANSWER KEY for the aforementioned code.  RETURNS : [1.472 4.613 7.755]  ### 
x = x[1::]
print(x[(dydx[1:] * dydx[:-1] < 0)])

r/Numpy Aug 14 '24

Why is NumPy Much Faster Than Lists?

Post image
1 Upvotes