.NET Framework - Large arrays in c#

Asked By Clive Tooth on 09-Feb-12 08:22 AM
This works ok...
const int n = 500000000;
int[] a = new int[n];
a[1] = 2;

This fails with "Exception of type 'System.OutOfMemoryException' was
thrown."...
const int n = 600000000;
int[] a = new int[n];
a[1] = 2;

There seems to be a limit of 2GB on any array.


My computer has 16GB of RAM.

I am using Visual C# 2010 Express, Windows 7 Home Premium Version 6.1
(build 7601: Service Pack 1)

Microsoft Visual Studio 2010
Version 10.0.30319.1 RTMRel
Microsoft .NET Framework
Version 4.0.30319 RTMRel

--
Clive Tooth




Marcel_Müller replied to Clive Tooth on 09-Feb-12 09:23 AM
No .NET object can exceed 2GB size. The allocator does not support this.
Because of some internal overhead, the practical maximum is somewhat less.


I would not recommend to allocate that large memory in one block for
several reasons.
Dealing with fragmentation in objects of that dimension will add no
significant overhead. But dealing with address space fragmentation will
be a serious point. There are BigArray implementations available.


This does not count, since the allocation is done in virtual memory. As
long as you do not access all these elements, most operating systems
will not associate physical memory with the address space. So the
important point is that you have a 64 bit platform and a 64 bit runtime.


Marcel
Peter Duniho replied to Marcel_Müller on 09-Feb-12 10:16 AM
Address space fragmentation is not an issue on 64-bit Windows.  Not for a

If the OP is trying to allocate such a large object on 32-bit Windows, then
yes...that is a bad idea.  it is not even possible, due to the 2GB
per-process limit on virtual address space, but of course even large
allocations that are possible would be a bad idea.

But presumably this is for a 64-bit process running on 64-bit Windows.  A
2GB allocation is really no big deal in that environment.  There is a
ginormous amount of virtual address space to work with in that case (to use
the technical term :) ).


Yes, and it _ought_ to work.  The .NET array types even support 64-bit
array lengths (LongLength property) in theory.  The problem is that you
cannot get .NET itself to allocate an array that large.

it is one of my pet peeves: 64-bit programs in .NET are not really fully
64-bits.  You get the advantages of the larger address space and higher
per-process memory limits, but individual objects are still limited in size
(as discussed here).  it is very annoying.  :(

Pete
Matt replied to Peter Duniho on 09-Feb-12 04:21 PM
s?
a
en
=A0A
se
.

That's correct, it was a conscious decision on the part of the
designers.
You can read one of the guy's motivations here:

http://blogs.msdn.com/b/joshwil/archive/2005/08/10/450202.aspx

It also shows a pretty nice way to get around the problem :)

ze

I agree, and likely it will be fixed in a future release. However,
realistically,
how often do you really want to be able to allocate 2G of memory in a
single
block? (And, as I type this, I can hear Bill Gates chuckling about his
256k
comment).

matt
Peter Duniho replied to Matt on 09-Feb-12 08:46 PM
Well, I do not think anyone was suggesting that .NET left large object
allocations out accidentally!  :)


I think all three options are hacks, relative to those people who actually
need large objects to be allocated.  Yes, they work.  And no, they are not
terribly complicated.  But I'd use the word "nice" only inasmuch as we
acknowledge that the need for the work-arounds in the first place is
unfortunate.


This, I doubt.  The article you referenced hedged a bit, but note that it
was written prior to the actual release of v2.0.  4.5 is about to come out,
and we still do not see any change in this particular issue.

I am not holding my breath.


would be exclude if that was the metric.

computing today in which handling large data sets is in fact useful.

Pete
Marcel Müller replied to Peter Duniho on 10-Feb-12 03:57 AM
Most likely it is an optimization.



It is not a question of the .NET version. It is only a question of the
VM. E.g. the Mono .NET VM does not have this restriction on 64 bit hosts.


Marcel
Peter Duniho replied to Marcel Müller on 10-Feb-12 12:49 PM
I do not understand that statement.  The CLR implementation for .NET is
exactly dependent on the version.


Mono is not even .NET.  it is a "completely different" (not counting the code
base it stated from) implementation of the CLR.  it is not relevant at all
to the question at hand, of whether the .NET implementation of the CLR will
ever support allocations > 2GB.

Pete
Arne Vajhøj replied to Peter Duniho on 14-Feb-12 09:19 PM
It does not matter much that LongLength exist. If it did not
it could be added.

The question is what to do with all the code that uses
Length.

And because arrays and collections interact in many ways
then making arrays "64 bit capable" would also require
collections to become "64 bit capable".

And then there is serialization between 32 and 64 bit systems.

It will not be a simple change.

Arne
Arne Vajhøj replied to Marcel Müller on 14-Feb-12 09:23 PM
AFAIK then Mono does not officially support Windows in 64 bit mode.

Arne
Peter Duniho replied to Arne Vajhøj on 14-Feb-12 11:24 PM
It "matters" inasmuch as it suggests someone was actually thinking about
array lengths > 2^32-1 elements, but did not go "all the way".  Which was
the point of what I wrote.


Such code is still limited to 2^32-1 elements.  But first, note that the
limit is 2GB objects.  You can have (many) fewer than 2^32-1 elements and
still take advantage of an object size larger than 2GB.

So you are not really talking about the same thing that the OP is asking
about.

Second, code that was limited to using the Length property would simply be
restricted to arrays with fewer than 2^32-1 elements.  How best to enforce
this is debatable, but IMHO the best way would be to throw an exception if
the Length property is used for an array where the length is larger than an
int value can store.

Most code would "just work".  After all, it is unlikely (not impossible, but
not likely) an array would wind up with too many elements for the Length
property to be valid if the code was not already aware of larger arrays.


No one said it would.  But it is well within the "normal" in terms of the
kind of development resources that are invested in .NET.  Lots of new
features in .NET and C# have appeared that require much more work than
enabling object allocation sizes larger than 2GB and array/collection sizes
greater than 2^32-1 (noting that the two are related, but independent
issues).

Pete
Clive Tooth replied to Clive Tooth on 16-Feb-12 08:15 AM
Thanks very much to all contributors to this thread. I have now
implemented my own 'BigArray' class that allows me to create the large
structures that I want to use.

--
Clive Tooth
Arne Vajhøj replied to Peter Duniho on 16-Feb-12 09:04 PM
Solving 1/10000 of the problem is not really solving anything.

Not only did they no go all the way - they made a step of
0.01 mm forward.



No.

But given that I did not quote anything from original post, then
that should not surprise anyone.


I completely agree that exception would be necessary.


No. Very likely.

All libraries that take arrays or collections as input.


MS could add this stuff in relative few hours.

But the effort by MS is just a tiny fraction of the problem.

Arne
Jan Kotas replied to Clive Tooth on 18-Apr-12 06:10 PM
.NET Framework 4.5 is adding support for 2GB+ arrays: http://msdn.microsoft.com/en-us/library/hh285054(v=vs.110).aspx
Jeff Johnson replied to Matt on 20-Apr-12 12:12 PM
A) It was 640K.

B) There is no real proof that Gates actually said this, but it lives on in
apocrypha....
Radu Cosoveanu replied to Clive Tooth on 03-May-12 09:19 AM
As you know, on 32 bits, the max addressing limit is 2 GB.
There are 2 solutions:
- using memory as bulk and addressing how much you want, if you really want to go on this direction, which unfortunatelly is not the good one.
- change solution using a storage system or a database.
Why do you need so much memory ?

Kind regards,
Radu Cosoveanu.