perf_to_profile fails with large (18gb) perf data about perf_data_converter

[JustinStitt]

I've located the problem (and proposed a fix):

In perf_to_profile_lib.cc there exists this invocation:

return perftools::RawPerfDataToProfiles(data.data(), data.length()/* data.length() is a size_t */, {},          
                                            sample_labels, options);

The type of data.length() or its alias data.size() is size_t. The problem lies within the perf_data_converter.cc RawPerfDataToProfiles() function which expects an integer for its raw_size parameter. This causes integer narrowing/truncation in which no data can be read past INT_MAX (2147483647) or ~2.14gb.

See the changes below:

diff --git a/src/perf_data_converter.cc b/src/perf_data_converter.cc               
index 2095e82..8775bd0 100644                                                      
--- a/src/perf_data_converter.cc                                                   
+++ b/src/perf_data_converter.cc                                                   
@@ -741,7 +741,7 @@ ProcessProfiles PerfDataProtoToProfiles(                       
 }                                                                                 
                                                                                   
 ProcessProfiles RawPerfDataToProfiles(                                            
-    const void* raw, const int raw_size,                                          
+    const void* raw, const uint64_t raw_size,                                     
     const std::map<string, string>& build_ids, const uint32 sample_labels,        
     const uint32 options, const std::map<Tid, string>& thread_types) {            
   quipper::PerfReader reader;                                                     
diff --git a/src/perf_data_converter.h b/src/perf_data_converter.h                 
index 8b12598..d94d53e 100644                                                      
--- a/src/perf_data_converter.h                                                    
+++ b/src/perf_data_converter.h                                                    
@@ -130,7 +130,7 @@ using ProcessProfiles = std::vector<std::unique_ptr<ProcessProfile>>;
 //                                                                                
 // Returns a vector of process profiles, empty if any error occurs.               
 extern ProcessProfiles RawPerfDataToProfiles(                                     
-    const void* raw, int raw_size,                                                
+    const void* raw, uint64_t raw_size,                                           
     const std::map<std::string, std::string>& build_ids,                          
     uint32 sample_labels = kNoLabels, uint32 options = kGroupByPids,              
     const std::map<uint32, std::string>& thread_types = {});                                                                                  

I opted for uint64_t as it is the widest integer type and is equivalent to size_t on 64-bit systems. I also identified instances elsewhere in the code base in which various size metrics were of type u64 or uint64_t (aliases). It makes sense to maintain some sense of consistency.

It should be noted that the current max protobuf size is 2gb. Currently, perf data is compressed (~80%) when it arrives in its final protobuf state which means this change is still worthwhile. The bottleneck is now the allowed max size of protobufs themselves. If the size of protobuf were to be increased then this change would become even more necessary. Currently, if we assume around 80% compression from perf data to profile data then our new maximum perf input size (assuming this change is implemented and there's a hard max of 2gb protobuf size) is ~10gb.

[shantuo]

It should be noted that the current max protobuf size is 2gb. Currently, perf data is compressed (~80%) when it arrives in its final protobuf state which means this change is still worthwhile. The bottleneck is now the allowed max size of protobufs themselves.

You're correct, this might not fix your problem if the data is way larger than the protobuf maximum. I'll submit a fix.

[shantuo]

Merged #124