Out of memory
Posted: 15 January 2015 08:43 PM   [ Ignore ]
Newbie
Rank
Total Posts:  17
Joined  2014-11-16

Hi Joost,

I am using your PERL code (found at http://www.wrds.us/index.php/tutorial/view/26) to download 10Ks and 10Qs.

Worked great til I began downloading the filings for 2014, where I experience the “out of memory” error.

Received some advice to read the files one line at a time rather than the entire file.

Still receive the error with the mod (which simply replaces the “for each” loop with a “while” loop).

Attached the code here.

Have any thoughts? 

Thank you!

File Attachments
Download_Filings_10K_10Q_step_2.pl.txt  (File Size: 2KB - Downloads: 162)
Profile
 
 
Posted: 15 January 2015 08:54 PM   [ Ignore ]   [ # 1 ]
Newbie
Rank
Total Posts:  17
Joined  2014-11-16

Hi Joost,

FYI, the size of the file which triggers the “out of memory” error is 422MB (2014 10-K for Entergy Arkansas, CIK 7323).

HTH!

Profile
 
 
Posted: 16 January 2015 09:56 AM   [ Ignore ]   [ # 2 ]
Administrator
Avatar
RankRankRankRank
Total Posts:  901
Joined  2011-09-19

hi wrkrbeee,

422 MB is quite large for a filing; would it make sense to do this filing manually (for example with your browser) and remove its id from the list of filings to download? I did a quick search on out of memory errors for the HTTP perl package, but that didn’t give me clues for suggestions.

best,

Joost

 Signature 

To reply/post new questions: Please use the group WRDS/SAS on Google Groups! http://groups.google.com/d/forum/wrdssas

Profile
 
 
Posted: 16 January 2015 04:12 PM   [ Ignore ]   [ # 3 ]
Newbie
Rank
Total Posts:  17
Joined  2014-11-16

Hi Joost,

Can you tell me why “$p” is necessary in line 53? Looks like “$response” already contains the data. In short, I’m wondering if we are duplicating the contents of a variable, which might create the “out of memory” problem.  Thanks!

Profile
 
 
Posted: 16 January 2015 06:25 PM   [ Ignore ]   [ # 4 ]
Administrator
Avatar
RankRankRankRank
Total Posts:  901
Joined  2011-09-19

hi wrkrbeee,

Yes, replacing $p with $response->content could work.

best,

Joost

 Signature 

To reply/post new questions: Please use the group WRDS/SAS on Google Groups! http://groups.google.com/d/forum/wrdssas

Profile