Begin PBS Prologue Thu Mar 5 15:54:10 EST 2009 End PBS Prologue Thu Mar 5 15:54:10 EST 2009 Warning: Permanently added the RSA host key for IP address '10.125.1.67' to the list of known hosts. Warning: Permanently added the RSA host key for IP address '10.125.1.67' to the list of known hosts. pvfs2-server: no process killed Warning: Permanently added the RSA host key for IP address '10.125.1.94' to the list of known hosts. Warning: Permanently added the RSA host key for IP address '10.125.1.94' to the list of known hosts. pvfs2-server: no process killed Warning: Permanently added the RSA host key for IP address '10.125.2.2' to the list of known hosts. Warning: Permanently added the RSA host key for IP address '10.125.2.2' to the list of known hosts. pvfs2-server: no process killed Warning: Permanently added the RSA host key for IP address '10.125.2.79' to the list of known hosts. Warning: Permanently added the RSA host key for IP address '10.125.2.79' to the list of known hosts. pvfs2-server: no process killed [D 15:54:15.402066] BMI_post_recv: addr: 3, offset: 0x2b156d202010, size: 65568, tag: 1 [D 15:54:15.402200] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 28, tag: 1 [D 15:54:15.402210] element 0: offset: 0x2b156d603010, size: 28 [D 15:54:29.091233] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 11 [D 15:54:29.091257] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 12 [D 15:54:29.091379] BMI_post_recv: addr: 3, offset: 0x2b156d603010, size: 65568, tag: 2 [D 15:54:29.091401] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 28, tag: 2 [D 15:54:29.091411] element 0: offset: 0x2b156d202010, size: 28 [D 15:54:29.091426] BMI_post_recv: addr: 20, offset: 0x2b156ca00010, size: 65568, tag: 3 [D 15:54:29.091441] BMI_post_sendunexpected_list: addr: 20, count: 1, total_size: 28, tag: 3 [D 15:54:29.091450] element 0: offset: 0x2b156ce01010, size: 28 [D 15:54:29.117146] BMI_post_recv: addr: 3, offset: 0x2b156ce01010, size: 16, tag: 4 [D 15:54:29.117164] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 28, tag: 4 [D 15:54:29.117172] element 0: offset: 0x2b156ca00010, size: 28 [D 15:54:29.126587] BMI_post_recv: addr: 20, offset: 0x2b156ca00010, size: 16, tag: 5 [D 15:54:29.126602] BMI_post_sendunexpected_list: addr: 20, count: 1, total_size: 28, tag: 5 [D 15:54:29.126611] element 0: offset: 0x2b156ce01010, size: 28 [D 15:54:29.126663] BMI_post_recv: addr: 3, offset: 0x2b156ce01010, size: 16, tag: 6 [D 15:54:29.126677] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 28, tag: 6 [D 15:54:29.126686] element 0: offset: 0x2b156ca00010, size: 28 [D 15:54:29.126743] BMI_post_recv: addr: 20, offset: 0x2b156ca00010, size: 16, tag: 7 [D 15:54:29.126757] BMI_post_sendunexpected_list: addr: 20, count: 1, total_size: 28, tag: 7 [D 15:54:29.126765] element 0: offset: 0x2b156ce01010, size: 28 [D 15:54:29.126820] BMI_post_recv: addr: 3, offset: 0x2b156ce01010, size: 16, tag: 8 [D 15:54:29.126834] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 48, tag: 8 [D 15:54:29.126842] element 0: offset: 0x2b156ca00010, size: 48 [D 15:54:29.126857] BMI_post_recv: addr: 20, offset: 0x2b156d202010, size: 16, tag: 9 [D 15:54:29.126872] BMI_post_sendunexpected_list: addr: 20, count: 1, total_size: 48, tag: 9 [D 15:54:29.126880] element 0: offset: 0x2b156d603010, size: 48 [D 15:54:29.126953] BMI_post_recv: addr: 3, offset: 0x2b156d603010, size: 16, tag: 10 [D 15:54:29.126968] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 48, tag: 10 [D 15:54:29.126976] element 0: offset: 0x2b156d202010, size: 48 [D 15:54:29.126987] BMI_post_recv: addr: 20, offset: 0x2b156ca00010, size: 16, tag: 11 [D 15:54:29.127002] BMI_post_sendunexpected_list: addr: 20, count: 1, total_size: 48, tag: 11 [D 15:54:29.127010] element 0: offset: 0x2b156ce01010, size: 48 time -p mpiexec -n 2 -npernode 1 /home/bradles/software/anl-io-test/bin/anl-io-test-mx -f pvfs2:/tmp/bradles-pav/mount/anl-io-data # Using mpi-io calls. [D 15:54:29.247162] BMI_post_recv: addr: 3, offset: 0x2b3c9634c010, size: 65568, tag: 1 [D 15:54:29.247367] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 28, tag: 1 [D 15:54:29.247389] element 0: offset: 0x2b3c9674d010, size: 28 [D 15:54:29.260809] BMI_post_recv: addr: 3, offset: 0x2b954a590010, size: 65568, tag: 1 [D 15:54:29.260986] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 28, tag: 1 [D 15:54:29.260995] element 0: offset: 0x2b954a991010, size: 28 [D 15:54:30.271204] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 11 [D 15:54:30.271231] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 12 [D 15:54:30.271356] BMI_post_recv: addr: 3, offset: 0x2b3c9674d010, size: 65568, tag: 2 [D 15:54:30.271387] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 28, tag: 2 [D 15:54:30.271408] element 0: offset: 0x2b3c9634c010, size: 28 [D 15:54:30.271434] BMI_post_recv: addr: 8, offset: 0x2b3c95b4a010, size: 65568, tag: 3 [D 15:54:30.271462] BMI_post_sendunexpected_list: addr: 8, count: 1, total_size: 28, tag: 3 [D 15:54:30.271482] element 0: offset: 0x2b3c95f4b010, size: 28 [D 15:54:30.271774] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 11 [D 15:54:30.271790] [BMI CONTROL]: BMI_set_info: set_info: 0 option: 12 [D 15:54:30.271909] BMI_post_recv: addr: 3, offset: 0x2b954a991010, size: 65568, tag: 2 [D 15:54:30.271929] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 28, tag: 2 [D 15:54:30.271936] element 0: offset: 0x2b954a590010, size: 28 [D 15:54:30.271945] BMI_post_recv: addr: 8, offset: 0x2b9549d8e010, size: 65568, tag: 3 [D 15:54:30.271958] BMI_post_sendunexpected_list: addr: 8, count: 1, total_size: 28, tag: 3 [D 15:54:30.271963] element 0: offset: 0x2b954a18f010, size: 28 [D 15:54:30.291292] BMI_post_recv: addr: 3, offset: 0x2b3c95f4b010, size: 8224, tag: 4 [D 15:54:30.291324] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 76, tag: 4 [D 15:54:30.291347] element 0: offset: 0x2b3c95b4a010, size: 76 [D 15:54:30.291296] BMI_post_recv: addr: 3, offset: 0x2b954a18f010, size: 8224, tag: 4 [D 15:54:30.291315] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 76, tag: 4 [D 15:54:30.291322] element 0: offset: 0x2b9549d8e010, size: 76 [D 15:54:30.296941] BMI_post_recv: addr: 3, offset: 0x2b3c95b4a010, size: 8224, tag: 5 [D 15:54:30.296971] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 76, tag: 5 [D 15:54:30.296991] element 0: offset: 0x2b3c95f4b010, size: 76 [D 15:54:30.297226] BMI_post_recv: addr: 3, offset: 0x2b3c95f4b010, size: 9284, tag: 6 [D 15:54:30.297252] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 56, tag: 6 [D 15:54:30.296944] BMI_post_recv: addr: 3, offset: 0x2b9549d8e010, size: 9284, tag: 5 [D 15:54:30.297271] element 0: offset: 0x2b3c95b4a010, size: 56 [D 15:54:30.296961] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 56, tag: 5 [D 15:54:30.296968] element 0: offset: 0x2b954a18f010, size: 56 [D 15:54:30.297421] BMI_post_recv: addr: 3, offset: 0x2b3c95b4a010, size: 8224, tag: 7 [D 15:54:30.297448] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 172, tag: 7 [D 15:54:30.297111] BMI_post_recv: addr: 3, offset: 0x2b954a18f010, size: 8224, tag: 6 [D 15:54:30.297124] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 172, tag: 6 [D 15:54:30.297130] element 0: offset: 0x2b9549d8e010, size: 172 [D 15:54:30.297486] element 0: offset: 0x2b3c95f4b010, size: 172 [E 15:59:30.551013] job_time_mgr_expire: job time out: cancelling bmi operation, job_id: 19. [D 15:59:30.551052] BMI_cancel: cancel id 32446400 [E 15:59:30.561240] Warning: msgpair failed to mx://node0252:0:3, will retry: Operation cancelled (possibly due to timeout) [D 15:59:30.561259] BMI_post_recv: addr: 3, offset: 0x2b954a18f010, size: 8224, tag: 7 [D 15:59:30.561279] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 172, tag: 7 [D 15:59:30.561287] element 0: offset: 0x2b9549d8e010, size: 172 [E 15:59:30.580444] job_time_mgr_expire: job time out: cancelling bmi operation, job_id: 22. [D 15:59:30.580494] BMI_cancel: cancel id 435386336 [E 15:59:30.590676] Warning: msgpair failed to mx://node0252:0:3, will retry: Operation cancelled (possibly due to timeout) [D 15:59:30.590699] BMI_post_recv: addr: 3, offset: 0x2b3c95b4a010, size: 8224, tag: 8 [D 15:59:30.590728] BMI_post_sendunexpected_list: addr: 3, count: 1, total_size: 172, tag: 8 [D 15:59:30.590746] element 0: offset: 0x2b3c95f4b010, size: 172 =>> PBS: job killed: walltime 603 exceeded limit 600