Why does a table with a defined constant in its index compute 10X slower?Why does array packing in Table behave like this?Why is building a table of function values so much slower than just plotting the function?Issue with very large lists in MathematicaWhy does `Table` returns values but `Plot` doesn't plot them?How to speed up computation of medoids?Why is CompilationTarget -> C slower than directly writing with C?ParallelTable much slower than Table on RandomReal with arbitrary precisionPart does not exist in table construction, but I don't explicitly index that highwhy Findroot slower with jacobian?Why does Table[] slow down exponentially with increasing length?

Bookshelves: the intruder

How would fantasy dwarves exist, realistically?

What should I wear to go and sign an employment contract?

Cycling to work - 30mile return

pwaS eht tirsf dna tasl setterl fo hace dorw

multicol package causes underfull hbox

Why are there five extra turns in tournament Magic?

Hotel booking: Why is Agoda much cheaper than booking.com?

Why is the S-duct intake on the Tu-154 uniquely oblong?

Why didn't Daenerys' advisers suggest assassinating Cersei?

What's is the easiest way to purchase a stock and hold it

Can I pay my credit card?

Have GoT's showrunners reacted to the poor reception of the final season?

Using `printf` to print variable containing `%` percent sign results in "bash: printf: `p': invalid format character"

How do you cope with rejection?

Gambler's Fallacy Dice

Why does Taylor’s series “work”?

Why would company (decision makers) wait for someone to retire, rather than lay them off, when their role is no longer needed?

Why does the U.S military use mercenaries?

Don't replace "|" with "(empty)" when generating slugs from title?

Why does the setUID bit work inconsistently?

How do I balance a campaign consisting of four kobold PCs?

How to get all possible paths in 0/1 matrix better way?

Can more than one instance of Bend Luck be applied to the same roll?



Why does a table with a defined constant in its index compute 10X slower?


Why does array packing in Table behave like this?Why is building a table of function values so much slower than just plotting the function?Issue with very large lists in MathematicaWhy does `Table` returns values but `Plot` doesn't plot them?How to speed up computation of medoids?Why is CompilationTarget -> C slower than directly writing with C?ParallelTable much slower than Table on RandomReal with arbitrary precisionPart does not exist in table construction, but I don't explicitly index that highwhy Findroot slower with jacobian?Why does Table[] slow down exponentially with increasing length?













7












$begingroup$


I need to do some iterative summations. Here is a minimum working example:



data = Table[RandomReal[], x, 1, 1000000];
(* Method 1 *)
Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]

(* Method 2, with constant index *)
m = 10;
Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]


And here are the outputs:




0.5625, 5.49936*10^6



9.28125, 5.49936*10^6




For some reason, using m=10 makes it much slower. I will need to do a bunch of m's, so this is the bottom of a larger nest.



What is a faster way to do this?



Late Edit:



Bonus question: How to optimize this one as well:



Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]









share|improve this question











$endgroup$
















    7












    $begingroup$


    I need to do some iterative summations. Here is a minimum working example:



    data = Table[RandomReal[], x, 1, 1000000];
    (* Method 1 *)
    Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]

    (* Method 2, with constant index *)
    m = 10;
    Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]


    And here are the outputs:




    0.5625, 5.49936*10^6



    9.28125, 5.49936*10^6




    For some reason, using m=10 makes it much slower. I will need to do a bunch of m's, so this is the bottom of a larger nest.



    What is a faster way to do this?



    Late Edit:



    Bonus question: How to optimize this one as well:



    Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]









    share|improve this question











    $endgroup$














      7












      7








      7


      2



      $begingroup$


      I need to do some iterative summations. Here is a minimum working example:



      data = Table[RandomReal[], x, 1, 1000000];
      (* Method 1 *)
      Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]

      (* Method 2, with constant index *)
      m = 10;
      Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]


      And here are the outputs:




      0.5625, 5.49936*10^6



      9.28125, 5.49936*10^6




      For some reason, using m=10 makes it much slower. I will need to do a bunch of m's, so this is the bottom of a larger nest.



      What is a faster way to do this?



      Late Edit:



      Bonus question: How to optimize this one as well:



      Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]









      share|improve this question











      $endgroup$




      I need to do some iterative summations. Here is a minimum working example:



      data = Table[RandomReal[], x, 1, 1000000];
      (* Method 1 *)
      Timing[Total[Table[Total[ Table[data[[i]], i, j, 10 + j]], j, 1, Length[data] - 5*10]]]

      (* Method 2, with constant index *)
      m = 10;
      Timing[Total[Table[Total[ Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*m]]]


      And here are the outputs:




      0.5625, 5.49936*10^6



      9.28125, 5.49936*10^6




      For some reason, using m=10 makes it much slower. I will need to do a bunch of m's, so this is the bottom of a larger nest.



      What is a faster way to do this?



      Late Edit:



      Bonus question: How to optimize this one as well:



      Timing[Total[Table[ (Total[ Table[data[[i]], i, j, m + j]])^2 , j, 1, Length[data] - 5*m]]]






      list-manipulation performance-tuning table






      share|improve this question















      share|improve this question













      share|improve this question




      share|improve this question








      edited 1 hour ago









      Carl Woll

      79.1k3102206




      79.1k3102206










      asked 4 hours ago









      axsvl77axsvl77

      405315




      405315




















          2 Answers
          2






          active

          oldest

          votes


















          9












          $begingroup$

          The problem lies mostly in the inner Table:



          Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]

          m = 10;
          Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]



          0.366407, 5.50276*10^6



          8.01738, 5.50276*10^6




          I think the reason is this:
          Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.



          You can help the JIT compiler by using With:



          With[m = 10,
          Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
          ]



          0.369601, 5.5049*10^6




          Addendum:



          By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):



          a = With[m = 10,
          Total[
          Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
          Length[data] - 5*m]]
          ]; // RepeatedTiming // First
          b = Total@ListCorrelate[ConstantArray[1., m + 1],
          data[[;; -50 + m - 1]]]; // RepeatedTiming // First
          c = Plus[
          Range[1., m].data[[1 ;; m]],
          (m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
          Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
          ]; // RepeatedTiming // First

          a == b == c



          0.28



          0.017



          0.0018



          True







          share|improve this answer











          $endgroup$




















            7












            $begingroup$

            You can use ListCorrelate:



            m=10;
            Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming



            0.017725, 5.50044*10^6




            Bonus question



            For the bonus question:



            data = RandomReal[1, 10^5];


            Your version:



            With[m = 10,
            Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
            ] //AbsoluteTiming



            0.448739, 3.11778*10^7




            Using ListCorrelate again:



            m = 10;
            #.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming



            0.018401, 3.11778*10^7







            share|improve this answer











            $endgroup$













              Your Answer








              StackExchange.ready(function()
              var channelOptions =
              tags: "".split(" "),
              id: "387"
              ;
              initTagRenderer("".split(" "), "".split(" "), channelOptions);

              StackExchange.using("externalEditor", function()
              // Have to fire editor after snippets, if snippets enabled
              if (StackExchange.settings.snippets.snippetsEnabled)
              StackExchange.using("snippets", function()
              createEditor();
              );

              else
              createEditor();

              );

              function createEditor()
              StackExchange.prepareEditor(
              heartbeatType: 'answer',
              autoActivateHeartbeat: false,
              convertImagesToLinks: false,
              noModals: true,
              showLowRepImageUploadWarning: true,
              reputationToPostImages: null,
              bindNavPrevention: true,
              postfix: "",
              imageUploader:
              brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
              contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
              allowUrls: true
              ,
              onDemand: true,
              discardSelector: ".discard-answer"
              ,immediatelyShowMarkdownHelp:true
              );



              );













              draft saved

              draft discarded


















              StackExchange.ready(
              function ()
              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198514%2fwhy-does-a-table-with-a-defined-constant-in-its-index-compute-10x-slower%23new-answer', 'question_page');

              );

              Post as a guest















              Required, but never shown

























              2 Answers
              2






              active

              oldest

              votes








              2 Answers
              2






              active

              oldest

              votes









              active

              oldest

              votes






              active

              oldest

              votes









              9












              $begingroup$

              The problem lies mostly in the inner Table:



              Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]

              m = 10;
              Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]



              0.366407, 5.50276*10^6



              8.01738, 5.50276*10^6




              I think the reason is this:
              Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.



              You can help the JIT compiler by using With:



              With[m = 10,
              Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
              ]



              0.369601, 5.5049*10^6




              Addendum:



              By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):



              a = With[m = 10,
              Total[
              Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
              Length[data] - 5*m]]
              ]; // RepeatedTiming // First
              b = Total@ListCorrelate[ConstantArray[1., m + 1],
              data[[;; -50 + m - 1]]]; // RepeatedTiming // First
              c = Plus[
              Range[1., m].data[[1 ;; m]],
              (m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
              Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
              ]; // RepeatedTiming // First

              a == b == c



              0.28



              0.017



              0.0018



              True







              share|improve this answer











              $endgroup$

















                9












                $begingroup$

                The problem lies mostly in the inner Table:



                Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]

                m = 10;
                Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]



                0.366407, 5.50276*10^6



                8.01738, 5.50276*10^6




                I think the reason is this:
                Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.



                You can help the JIT compiler by using With:



                With[m = 10,
                Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
                ]



                0.369601, 5.5049*10^6




                Addendum:



                By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):



                a = With[m = 10,
                Total[
                Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
                Length[data] - 5*m]]
                ]; // RepeatedTiming // First
                b = Total@ListCorrelate[ConstantArray[1., m + 1],
                data[[;; -50 + m - 1]]]; // RepeatedTiming // First
                c = Plus[
                Range[1., m].data[[1 ;; m]],
                (m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
                Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
                ]; // RepeatedTiming // First

                a == b == c



                0.28



                0.017



                0.0018



                True







                share|improve this answer











                $endgroup$















                  9












                  9








                  9





                  $begingroup$

                  The problem lies mostly in the inner Table:



                  Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]

                  m = 10;
                  Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]



                  0.366407, 5.50276*10^6



                  8.01738, 5.50276*10^6




                  I think the reason is this:
                  Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.



                  You can help the JIT compiler by using With:



                  With[m = 10,
                  Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
                  ]



                  0.369601, 5.5049*10^6




                  Addendum:



                  By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):



                  a = With[m = 10,
                  Total[
                  Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
                  Length[data] - 5*m]]
                  ]; // RepeatedTiming // First
                  b = Total@ListCorrelate[ConstantArray[1., m + 1],
                  data[[;; -50 + m - 1]]]; // RepeatedTiming // First
                  c = Plus[
                  Range[1., m].data[[1 ;; m]],
                  (m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
                  Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
                  ]; // RepeatedTiming // First

                  a == b == c



                  0.28



                  0.017



                  0.0018



                  True







                  share|improve this answer











                  $endgroup$



                  The problem lies mostly in the inner Table:



                  Timing[Total[Table[Total[Table[data[[i]], i, j, 10 + j]], j, 1,Length[data] - 5*10]]]

                  m = 10;
                  Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1, Length[data] - 5*10]]]



                  0.366407, 5.50276*10^6



                  8.01738, 5.50276*10^6




                  I think the reason is this:
                  Because the global variable m could theoretically change its value during the computions, the body of the outer table cannot be compiled (without calls to MainEvaluate). At least, the JIT compiler does not analyze the body of the outer loop thoroughly enough to decide that m won't change.



                  You can help the JIT compiler by using With:



                  With[m = 10,
                  Timing[Total[Table[Total[Table[data[[i]], i, j, m + j]], j, 1,Length[data] - 5*m]]]
                  ]



                  0.369601, 5.5049*10^6




                  Addendum:



                  By focusing on the post's title, I have completely overlooked the question on how to make it faster. Here is my proposal (c) vs. the OP's one (a) and Carl's (b):



                  a = With[m = 10,
                  Total[
                  Table[Total[Table[data[[i]], i, j, m + j]], j, 1,
                  Length[data] - 5*m]]
                  ]; // RepeatedTiming // First
                  b = Total@ListCorrelate[ConstantArray[1., m + 1],
                  data[[;; -50 + m - 1]]]; // RepeatedTiming // First
                  c = Plus[
                  Range[1., m].data[[1 ;; m]],
                  (m + 1) Total[data[[m + 1 ;; -5*m - 1]]],
                  Range[N@m, 1., -1].data[[-5 m ;; -4 m - 1]]
                  ]; // RepeatedTiming // First

                  a == b == c



                  0.28



                  0.017



                  0.0018



                  True








                  share|improve this answer














                  share|improve this answer



                  share|improve this answer








                  edited 2 hours ago

























                  answered 4 hours ago









                  Henrik SchumacherHenrik Schumacher

                  62.4k586173




                  62.4k586173





















                      7












                      $begingroup$

                      You can use ListCorrelate:



                      m=10;
                      Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming



                      0.017725, 5.50044*10^6




                      Bonus question



                      For the bonus question:



                      data = RandomReal[1, 10^5];


                      Your version:



                      With[m = 10,
                      Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
                      ] //AbsoluteTiming



                      0.448739, 3.11778*10^7




                      Using ListCorrelate again:



                      m = 10;
                      #.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming



                      0.018401, 3.11778*10^7







                      share|improve this answer











                      $endgroup$

















                        7












                        $begingroup$

                        You can use ListCorrelate:



                        m=10;
                        Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming



                        0.017725, 5.50044*10^6




                        Bonus question



                        For the bonus question:



                        data = RandomReal[1, 10^5];


                        Your version:



                        With[m = 10,
                        Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
                        ] //AbsoluteTiming



                        0.448739, 3.11778*10^7




                        Using ListCorrelate again:



                        m = 10;
                        #.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming



                        0.018401, 3.11778*10^7







                        share|improve this answer











                        $endgroup$















                          7












                          7








                          7





                          $begingroup$

                          You can use ListCorrelate:



                          m=10;
                          Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming



                          0.017725, 5.50044*10^6




                          Bonus question



                          For the bonus question:



                          data = RandomReal[1, 10^5];


                          Your version:



                          With[m = 10,
                          Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
                          ] //AbsoluteTiming



                          0.448739, 3.11778*10^7




                          Using ListCorrelate again:



                          m = 10;
                          #.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming



                          0.018401, 3.11778*10^7







                          share|improve this answer











                          $endgroup$



                          You can use ListCorrelate:



                          m=10;
                          Total @ ListCorrelate[ConstantArray[1,m+1], data[[;;-4 m-1]]] //AbsoluteTiming



                          0.017725, 5.50044*10^6




                          Bonus question



                          For the bonus question:



                          data = RandomReal[1, 10^5];


                          Your version:



                          With[m = 10,
                          Total[Table[(Total[Table[data[[i]],i,j,m+j]])^2,j,1,Length[data]-5*m]]
                          ] //AbsoluteTiming



                          0.448739, 3.11778*10^7




                          Using ListCorrelate again:



                          m = 10;
                          #.#& @ ListCorrelate[ConstantArray[1, m+1], data[[ ;; -4 m - 1]]] //AbsoluteTiming



                          0.018401, 3.11778*10^7








                          share|improve this answer














                          share|improve this answer



                          share|improve this answer








                          edited 1 hour ago

























                          answered 4 hours ago









                          Carl WollCarl Woll

                          79.1k3102206




                          79.1k3102206



























                              draft saved

                              draft discarded
















































                              Thanks for contributing an answer to Mathematica Stack Exchange!


                              • Please be sure to answer the question. Provide details and share your research!

                              But avoid


                              • Asking for help, clarification, or responding to other answers.

                              • Making statements based on opinion; back them up with references or personal experience.

                              Use MathJax to format equations. MathJax reference.


                              To learn more, see our tips on writing great answers.




                              draft saved


                              draft discarded














                              StackExchange.ready(
                              function ()
                              StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fmathematica.stackexchange.com%2fquestions%2f198514%2fwhy-does-a-table-with-a-defined-constant-in-its-index-compute-10x-slower%23new-answer', 'question_page');

                              );

                              Post as a guest















                              Required, but never shown





















































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown

































                              Required, but never shown














                              Required, but never shown












                              Required, but never shown







                              Required, but never shown







                              Popular posts from this blog

                              Siegen Nawigatsjuun

                              Log på Navigationsmenu

                              Log på Navigationsmenu