Computing elements of a 1000 x 60 matrix exhausts RAMWhat is a Mathematica packed array?Paging RAM in case of memory shortage issueHeavy duty operations, RAM, and ReadyboostEfficient calculation of diagonal matrix elementsHow to know each variable used how much RAMReplacing elements of a matrixHow to clear RAM memory in a running code?How does this code behave with more than 32GiB RAM?Replace diagonal elements in sparse matrixSolution list from “Solve” too large for my RAM spaceLimit the amount of RAM Mathematica may access?

Why is this python script running in background consuming 100 % CPU?

Negative impact of having the launch pad away from the Equator

Passport queue length in UK in relation to arrival method

What is the required burn to keep a satellite at a Lagrangian point?

What is the winged creature on the back of the Mordenkainen's Tome of Foes book?

Is the default 512 byte physical sector size appropriate for SSD disks under Linux?

why "American-born", not "America-born"?

Computing elements of a 1000 x 60 matrix exhausts RAM

How to tease a romance without a cat and mouse chase?

Why is this integration method not valid?

Is it normal to "extract a paper" from a master thesis?

If a character has cast the Fly spell on themselves, can they "hand off" to the Levitate spell without interruption?

How does the Earth's center produce heat?

size of pointers and architecture

Is there a word for pant sleeves?

Wifi light switch needs neutral wire. Why? AND Can that wire be a skinny one?

Why the work done is positive when bringing 2 opposite charges together?

Caught with my phone during an exam

Illustrating that universal optimality is stronger than sphere packing

Way of refund if scammed?

amsmath: How can I use the equation numbering and label manually and anywhere?

Writing "hahaha" versus describing the laugh

Keeping the dodos out of the field

How to become an Editorial board member?



Computing elements of a 1000 x 60 matrix exhausts RAM


What is a Mathematica packed array?Paging RAM in case of memory shortage issueHeavy duty operations, RAM, and ReadyboostEfficient calculation of diagonal matrix elementsHow to know each variable used how much RAMReplacing elements of a matrixHow to clear RAM memory in a running code?How does this code behave with more than 32GiB RAM?Replace diagonal elements in sparse matrixSolution list from “Solve” too large for my RAM spaceLimit the amount of RAM Mathematica may access?













5












$begingroup$


I am trying to compute a 1000 x 60 matrix or list of lists (and ideally this should go up to 1000 x 500 or 1000 x 1000).



Each element is the result of a FindRoot operation, so I make my list by doing



Table[Flatten[h /. FindRoot[h == F[h, b, g], b, 1, 1000, g, 1, 60)


but 16GB of RAM are filled up. I think I should be able to hold list of lists much bigger than that, so probably using Table with FindRoot is causing Mathematica to store a lot of undeeded stuff in memory.



Here is the code:



ι[m_, n_] := Binomial[n, n*(1 - m)/2]*2^(-n);
f[m_, h_, b_, g_, n_] := (h*m + g/2*m^2) +
1/(n*b)*Log[ι[m, n]];
μ[m_, h_, b_, g_, n_] :=
Exp[b*n*f[m, h, b, g, n] + b*n*(-h + g/2)]/
Sum[Exp[b*n*f[x, h, b, g, n] + b*n*(-h + g/2)], x, -1 + 2/n,
1 - 2/n, 2/n];
moment[h_, x_, b_, g_, n_] := Sum[m^x*μ[m, h, b, g, n], m, -1 + 2/n, 1 - 2/n, 2/n];
var[h_, b_, g_, n_] := moment[h, 2, b, g, n] - moment[h, 1, b, g, n]^2;
cov[h_, b_, g_, n_] := moment[h, 3, b, g, n] - moment[h, 1, b, g,n]*moment[h, 2, b, g, n];
F[h_,b_,g_,n_]:= -d*b*(cov[h, b, gg, n] +
2 var[h, b, gg, n]);
n = 100;
d = 0.9;

glist = Table[g, g, 0.4, 1, 0.01];
blist = Table[b, b, 1.1, 10.1, 0.01];

heatdata = Table[
Flatten[h /.
FindRoot[
h == F[h,b,g,n], h, -0.01]][[1]]
, b, blist, g, glist];









share|improve this question











$endgroup$











  • $begingroup$
    Please show a complete minimal example that reproduces the problem.
    $endgroup$
    – Szabolcs
    17 hours ago






  • 1




    $begingroup$
    If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
    $endgroup$
    – Szabolcs
    17 hours ago






  • 1




    $begingroup$
    One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
    $endgroup$
    – Szabolcs
    17 hours ago










  • $begingroup$
    Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
    $endgroup$
    – Three Diag
    17 hours ago















5












$begingroup$


I am trying to compute a 1000 x 60 matrix or list of lists (and ideally this should go up to 1000 x 500 or 1000 x 1000).



Each element is the result of a FindRoot operation, so I make my list by doing



Table[Flatten[h /. FindRoot[h == F[h, b, g], b, 1, 1000, g, 1, 60)


but 16GB of RAM are filled up. I think I should be able to hold list of lists much bigger than that, so probably using Table with FindRoot is causing Mathematica to store a lot of undeeded stuff in memory.



Here is the code:



ι[m_, n_] := Binomial[n, n*(1 - m)/2]*2^(-n);
f[m_, h_, b_, g_, n_] := (h*m + g/2*m^2) +
1/(n*b)*Log[ι[m, n]];
μ[m_, h_, b_, g_, n_] :=
Exp[b*n*f[m, h, b, g, n] + b*n*(-h + g/2)]/
Sum[Exp[b*n*f[x, h, b, g, n] + b*n*(-h + g/2)], x, -1 + 2/n,
1 - 2/n, 2/n];
moment[h_, x_, b_, g_, n_] := Sum[m^x*μ[m, h, b, g, n], m, -1 + 2/n, 1 - 2/n, 2/n];
var[h_, b_, g_, n_] := moment[h, 2, b, g, n] - moment[h, 1, b, g, n]^2;
cov[h_, b_, g_, n_] := moment[h, 3, b, g, n] - moment[h, 1, b, g,n]*moment[h, 2, b, g, n];
F[h_,b_,g_,n_]:= -d*b*(cov[h, b, gg, n] +
2 var[h, b, gg, n]);
n = 100;
d = 0.9;

glist = Table[g, g, 0.4, 1, 0.01];
blist = Table[b, b, 1.1, 10.1, 0.01];

heatdata = Table[
Flatten[h /.
FindRoot[
h == F[h,b,g,n], h, -0.01]][[1]]
, b, blist, g, glist];









share|improve this question











$endgroup$











  • $begingroup$
    Please show a complete minimal example that reproduces the problem.
    $endgroup$
    – Szabolcs
    17 hours ago






  • 1




    $begingroup$
    If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
    $endgroup$
    – Szabolcs
    17 hours ago






  • 1




    $begingroup$
    One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
    $endgroup$
    – Szabolcs
    17 hours ago










  • $begingroup$
    Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
    $endgroup$
    – Three Diag
    17 hours ago













5












5








5


1



$begingroup$


I am trying to compute a 1000 x 60 matrix or list of lists (and ideally this should go up to 1000 x 500 or 1000 x 1000).



Each element is the result of a FindRoot operation, so I make my list by doing



Table[Flatten[h /. FindRoot[h == F[h, b, g], b, 1, 1000, g, 1, 60)


but 16GB of RAM are filled up. I think I should be able to hold list of lists much bigger than that, so probably using Table with FindRoot is causing Mathematica to store a lot of undeeded stuff in memory.



Here is the code:



ι[m_, n_] := Binomial[n, n*(1 - m)/2]*2^(-n);
f[m_, h_, b_, g_, n_] := (h*m + g/2*m^2) +
1/(n*b)*Log[ι[m, n]];
μ[m_, h_, b_, g_, n_] :=
Exp[b*n*f[m, h, b, g, n] + b*n*(-h + g/2)]/
Sum[Exp[b*n*f[x, h, b, g, n] + b*n*(-h + g/2)], x, -1 + 2/n,
1 - 2/n, 2/n];
moment[h_, x_, b_, g_, n_] := Sum[m^x*μ[m, h, b, g, n], m, -1 + 2/n, 1 - 2/n, 2/n];
var[h_, b_, g_, n_] := moment[h, 2, b, g, n] - moment[h, 1, b, g, n]^2;
cov[h_, b_, g_, n_] := moment[h, 3, b, g, n] - moment[h, 1, b, g,n]*moment[h, 2, b, g, n];
F[h_,b_,g_,n_]:= -d*b*(cov[h, b, gg, n] +
2 var[h, b, gg, n]);
n = 100;
d = 0.9;

glist = Table[g, g, 0.4, 1, 0.01];
blist = Table[b, b, 1.1, 10.1, 0.01];

heatdata = Table[
Flatten[h /.
FindRoot[
h == F[h,b,g,n], h, -0.01]][[1]]
, b, blist, g, glist];









share|improve this question











$endgroup$




I am trying to compute a 1000 x 60 matrix or list of lists (and ideally this should go up to 1000 x 500 or 1000 x 1000).



Each element is the result of a FindRoot operation, so I make my list by doing



Table[Flatten[h /. FindRoot[h == F[h, b, g], b, 1, 1000, g, 1, 60)


but 16GB of RAM are filled up. I think I should be able to hold list of lists much bigger than that, so probably using Table with FindRoot is causing Mathematica to store a lot of undeeded stuff in memory.



Here is the code:



ι[m_, n_] := Binomial[n, n*(1 - m)/2]*2^(-n);
f[m_, h_, b_, g_, n_] := (h*m + g/2*m^2) +
1/(n*b)*Log[ι[m, n]];
μ[m_, h_, b_, g_, n_] :=
Exp[b*n*f[m, h, b, g, n] + b*n*(-h + g/2)]/
Sum[Exp[b*n*f[x, h, b, g, n] + b*n*(-h + g/2)], x, -1 + 2/n,
1 - 2/n, 2/n];
moment[h_, x_, b_, g_, n_] := Sum[m^x*μ[m, h, b, g, n], m, -1 + 2/n, 1 - 2/n, 2/n];
var[h_, b_, g_, n_] := moment[h, 2, b, g, n] - moment[h, 1, b, g, n]^2;
cov[h_, b_, g_, n_] := moment[h, 3, b, g, n] - moment[h, 1, b, g,n]*moment[h, 2, b, g, n];
F[h_,b_,g_,n_]:= -d*b*(cov[h, b, gg, n] +
2 var[h, b, gg, n]);
n = 100;
d = 0.9;

glist = Table[g, g, 0.4, 1, 0.01];
blist = Table[b, b, 1.1, 10.1, 0.01];

heatdata = Table[
Flatten[h /.
FindRoot[
h == F[h,b,g,n], h, -0.01]][[1]]
, b, blist, g, glist];






performance-tuning memory






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited 2 hours ago









m_goldberg

90.2k873203




90.2k873203










asked 18 hours ago









Three DiagThree Diag

346110




346110











  • $begingroup$
    Please show a complete minimal example that reproduces the problem.
    $endgroup$
    – Szabolcs
    17 hours ago






  • 1




    $begingroup$
    If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
    $endgroup$
    – Szabolcs
    17 hours ago






  • 1




    $begingroup$
    One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
    $endgroup$
    – Szabolcs
    17 hours ago










  • $begingroup$
    Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
    $endgroup$
    – Three Diag
    17 hours ago
















  • $begingroup$
    Please show a complete minimal example that reproduces the problem.
    $endgroup$
    – Szabolcs
    17 hours ago






  • 1




    $begingroup$
    If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
    $endgroup$
    – Szabolcs
    17 hours ago






  • 1




    $begingroup$
    One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
    $endgroup$
    – Szabolcs
    17 hours ago










  • $begingroup$
    Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
    $endgroup$
    – Three Diag
    17 hours ago















$begingroup$
Please show a complete minimal example that reproduces the problem.
$endgroup$
– Szabolcs
17 hours ago




$begingroup$
Please show a complete minimal example that reproduces the problem.
$endgroup$
– Szabolcs
17 hours ago




1




1




$begingroup$
If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
$endgroup$
– Szabolcs
17 hours ago




$begingroup$
If it's a lot of code, that would be your actual code, not a minimal example. Please make an effort to track down the cause of the problem, and construct a small example that illustrates the problem. See here for guidance: mathematica.meta.stackexchange.com/q/2126/12
$endgroup$
– Szabolcs
17 hours ago




1




1




$begingroup$
One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
$endgroup$
– Szabolcs
17 hours ago




$begingroup$
One possible issue is the memoization. Did you check how many values are actually saved? If you are working with floating point numbers, it may be the thing that eats up the memory.
$endgroup$
– Szabolcs
17 hours ago












$begingroup$
Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
$endgroup$
– Three Diag
17 hours ago




$begingroup$
Thanks, I've added the code as it isn't really that long. I've removed the memoization and I'm looking to see if this works now (the computation does take a while to run).
$endgroup$
– Three Diag
17 hours ago










1 Answer
1






active

oldest

votes


















19












$begingroup$

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.






share|improve this answer











$endgroup$












  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    3 hours ago











Your Answer








StackExchange.ready(function()
var channelOptions =
tags: "".split(" "),
id: "387"
;
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function()
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled)
StackExchange.using("snippets", function()
createEditor();
);

else
createEditor();

);

function createEditor()
StackExchange.prepareEditor(
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader:
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
,
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
);



);













draft saved

draft discarded


















StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198651%2fcomputing-elements-of-a-1000-x-60-matrix-exhausts-ram%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









19












$begingroup$

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.






share|improve this answer











$endgroup$












  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    3 hours ago















19












$begingroup$

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.






share|improve this answer











$endgroup$












  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    3 hours ago













19












19








19





$begingroup$

Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.






share|improve this answer











$endgroup$



Your function F is implemented really, really inefficiently. By quite simple means and in the proposed situation, it can be sped up by a factor of 20000. The key is to start with calculations in machine precision as early as possible and to store frequently used data in packed arrays.



n = 100;
mlist = Range[-1. + 2/n, 1. - 2/n, 2./n];
m2list = mlist^2;
m3list = mlist^3;
logiotalist = Log[Binomial[n, n*(1 - mlist)/2]*2^(-n)];

d = 0.9;
glist = Range[0.4, 1, 0.01];
blist = Range[1.1, 10.1, 0.01];

ClearAll[F];
F[h_?NumericQ, b_, g_] :=
Module[var, cov, explist, μlist, mom1, mom2, mom3,
explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];
μlist = explist/Total[explist];
mom1 = μlist.mlist;
mom2 = μlist.m2list;
mom3 = μlist.m3list;
var = Subtract[mom2, mom1 mom1];
cov = Subtract[mom3, mom1 mom2];
(-d b) (cov + 2. var)
];


Just a quick test for precision and speed:



t1, r1 = F[0.1, blist[[1]], glist[[1]], n] // RepeatedTiming;
t2, r2 = Fnew[0.1, blist[[1]], glist[[1]]] // RepeatedTiming;
Abs[r1 - r2]/r1
t1/t2



-1.32375*10^-14



2.1*10^4




Now the parallelized solve loop requires about 10 seconds on my Quad Core Haswell CPU:



ParallelEvaluate[Off[General::munfl]];
heatdata = Developer`ToPackedArray[
ParallelTable[
Block[h0, h,
h0 = -0.01;
Developer`ToPackedArray[
Table[
h0 = h /. FindRoot[h == Fnew[h, b, g], h, h0],
b, blist]
]
],
g, glist]
]; // AbsoluteTiming // First



10.072




Memory considerations



You also see: Limited amount of RAM is not an issue here. That must have been caused by excessive memoziation. For the timing, it is crucial how information is stored and retrieved.
Storing computed values in a packed array for retrieving them later is significantly more efficient than memoization. Memoization into DownValues uses a complex data structure such as a hash table at its backend, and this data structure has certain overhead. In contrast, a packed array represents basically a connected block of physical memory, accompanied by some bytes of meta information (array dimensions and maybe some row pointers). Moreover, computation with data stored in packed arrays can take advantage of vectorization, which is most crucially employed in the following line:



explist = Exp[(b n h) mlist + (b n g/2) m2list + logiotalist + b n (-h + g/2)];


Remark on precision



Finally, I have to note that there is numerical underflow occurring in the course of the computation. This is probably caused by calling Exp with negative numbers of oversized absolute value. I decided to turn off the warning message, but this may lead to a significant loss of precision. So use with care. If one wants to do it correctly, one should investigate this further and apply, e.g. Clip or Threshold.







share|improve this answer














share|improve this answer



share|improve this answer








edited 8 hours ago

























answered 16 hours ago









Henrik SchumacherHenrik Schumacher

62.8k587176




62.8k587176











  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    3 hours ago
















  • $begingroup$
    Thanks for this very helpful and informative answer!
    $endgroup$
    – Three Diag
    3 hours ago















$begingroup$
Thanks for this very helpful and informative answer!
$endgroup$
– Three Diag
3 hours ago




$begingroup$
Thanks for this very helpful and informative answer!
$endgroup$
– Three Diag
3 hours ago

















draft saved

draft discarded
















































Thanks for contributing an answer to Mathematica Stack Exchange!


  • Please be sure to answer the question. Provide details and share your research!

But avoid


  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.

Use MathJax to format equations. MathJax reference.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function ()
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198651%2fcomputing-elements-of-a-1000-x-60-matrix-exhausts-ram%23new-answer', 'question_page');

);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

Log på Navigationsmenu

Creating second map without labels using QGIS?How to lock map labels for inset map in Print Composer?How to Force the Showing of Labels of a Vector File in QGISQGIS Valmiera, Labels only show for part of polygonsRemoving duplicate point labels in QGISLabeling every feature using QGIS?Show labels for point features outside map canvasAbbreviate Road Labels in QGIS only when requiredExporting map from composer in QGIS - text labels have moved in output?How to make sure labels in qgis turn up in layout map?Writing label expression with ArcMap and If then Statement?

Detroit Tigers Spis treści Historia | Skład zespołu | Sukcesy | Członkowie Baseball Hall of Fame | Zastrzeżone numery | Przypisy | Menu nawigacyjneEncyclopedia of Detroit - Detroit TigersTigers Stadium, Detroit, MITigers Timeline 1900sDetroit Tigers Team History & EncyclopediaTigers Timeline 1910s1935 World Series1945 World Series1945 World Series1984 World SeriesComerica Park, Detroit, MI2006 World Series2012 World SeriesDetroit Tigers 40-Man RosterDetroit Tigers Coaching StaffTigers Hall of FamersTigers Retired Numberse