perforce / p4transfer Goto Github PK

I've configured P4Transfer and started the first transfer. The first changelist generated the following error:

2023-01-26 04:01:01,675:P4Transfer:INFO: Logging to file: /home/perforce/log-P4Transfer-20230126040101.log
2023-01-26 04:01:33,885:P4Transfer:INFO: Transferring 1000 changes
2023-01-26 04:01:34,831:P4Transfer:INFO: Syncing 1000 changes
2023-01-26 04:01:34,831:P4Transfer:INFO: Finding change sizes
2023-01-26 04:01:40,991:P4Transfer:INFO: Syncing filerevs 59739, size 2.8 GB
2023-01-26 04:01:41,025:P4Transfer:WARNING: warning result: ['//p4transfer-client/... - file(s) not opened on this client.']
2023-01-26 04:01:41,025:P4Transfer:INFO: Processing change: 102493, files 18887, size 1.1 GB "Rename //<redacted>/... To //<redacted>/..."
2023-01-26 04:01:47,720:P4Transfer:INFO: Synced 1/1000 changes, files 5353/59739 (9.0 %), size 508.6 MB/2.8 GB (17.8 %)
2023-01-26 04:01:47,781:P4Transfer:INFO: Synced 1/1000 changes, files 15879/59739 (26.6 %), size 986.5 MB/2.8 GB (34.6 %)
2023-01-26 04:02:20,480:P4Transfer:ERROR: 'utf-8' codec can't decode byte 0xb5 in position 341: invalid start byte
Traceback (most recent call last):
  File "/perforce/transfer/p4transfer/P4Transfer.py", line 2588, in replicate
    num_changes = self.replicate_changes()
  File "/perforce/transfer/p4transfer/P4Transfer.py", line 2404, in replicate_changes
    fileRevs, specialMoveRevs, srcFileLogs = self.source.getChange(change['change'])
  File "/perforce/transfer/p4transfer/P4Transfer.py", line 1249, in getChange
    chRev.updateDigest()
  File "/perforce/transfer/p4transfer/P4Transfer.py", line 584, in updateDigest
    self.fileSize, self.digest = getKTextDigest(self.fixedLocalFile)
  File "/perforce/transfer/p4transfer/P4Transfer.py", line 479, in getKTextDigest
    contents = contents.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 341: invalid start byte
2023-01-26 04:02:20,481:P4Transfer:INFO: Sleeping on error for 60 minutes

What are my next steps in troubleshooting this issue?

A key error can happen if a change has no revisions:

Traceback (most recent call last):
  File "/home/user/p4transfer/P4Transfer.py", line 2579, in replicate
    num_changes = self.replicate_changes()
  File "/home/user/p4transfer/P4Transfer.py", line 2396, in replicate_changes
    targetChange = self.target.replicateChange(fileRevs, specialMoveRevs, srcFileLogs, change, self.source.p4.port)
  File "/home/user/p4transfer/P4Transfer.py", line 1574, in replicateChange
    self.validateSubmittedChange(newChangeId, fileRevs)
  File "/home/user/p4transfer/P4Transfer.py", line 1624, in validateSubmittedChange
    for (n, rev) in enumerate(change['rev']):
KeyError: 'rev'

I added a bit of debugging around this for the corresponding change object (returned from self.p4cmd('describe', '-s', newChangeId)[0]). The dictionary looks like:

{
    'change': '1234',
    'user': 'someuser',
    'client': 'migration-target',
    'time': '1594805092',
    'desc': '(the changelist description)',
    'status': 'submitted',
    'changeType': 'public',
    'path': ''
}

The destination depot is a graph depot, which makes the change description different.

Attempting a stream transfer, it seems that if the file name has a # in the name it is interpreted as a version number.
Python 3.11.4 on windows.
Log Tail:
2023-06-21 11:35:43,506:P4Transfer:ERROR: [P4#run] Errors during command execution( "p4 sync -k e:\p4transfer\XXXX_SW_stream\master\PC\Test\Reference Documents\Master PDFs\XXXXSendProt BBBBB-#2 Protocol Sent to CCCC.pdf" )

[Error]: "Invalid revision number '2 Protocol Sent to CCCC.pdf'."

Traceback (most recent call last):
File "C:\Users\Administrator\Documents\p4transfer\P4Transfer.py", line 2632, in replicate
num_changes = self.replicate_changes()
^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Documents\p4transfer\P4Transfer.py", line 2449, in replicate_changes
targetChange = self.target.replicateChange(fileRevs, specialMoveRevs, srcFileLogs, change, self.source.p4.port)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\Administrator\Documents\p4transfer\P4Transfer.py", line 1556, in replicateChange
self.processChangeRevs(fileRevs, specialMoveRevs, srcFileLogs)
File "C:\Users\Administrator\Documents\p4transfer\P4Transfer.py", line 1494, in processChangeRevs
self.replicateBranch(f, dirty=False)
File "C:\Users\Administrator\Documents\p4transfer\P4Transfer.py", line 1872, in replicateBranch
self.p4cmd('sync', '-k', file.fixedLocalFile)
File "C:\Users\Administrator\Documents\p4transfer\P4Transfer.py", line 1024, in p4cmd
output = self.p4.run(args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Python311\Lib\site-packages\P4.py", line 617, in run
raise e
File "C:\Python311\Lib\site-packages\P4.py", line 611, in run
result = P4API.P4Adapter.run(self, *flatArgs)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
P4.P4Exception: [P4#run] Errors during command execution( "p4 sync -k e:\p4transfer\XXXX_SW_stream\master\PC\Test\Reference Documents\Master PDFs\XXXXSendProt BBBBB-#2 Protocol Sent to CCCC.pdf" )

[Error]: "Invalid revision number '2 Protocol Sent to CCCC.pdf'."

I have noticed there can be issues when files are copied from one depot location to another and edited in the same change list before they were submitted (sometimes referred to as an "edigrate"). In this case, the added file was copied from another depot location and then edited before submission:

2021-10-29 03:27:01,402:P4Transfer:ERROR: Replication failure: src/target content differences found
src:rev=1 action=add type=text size=18847 digest=1143C61AAA76935548760CACCE135A60 depotFile=//Foo/Bar.cs
targ:rev=1 action=add type=text size=18859 digest=DC71D925CE536B7E669FA4FF7117EA25 depotFile=//Foo/Bar.cs

In this case should the source file be re-synced to account for the edit during the copy / add?

Transferring from a depot that has p4ignore files can interfere with target changelist processing.

2021-10-17 21:00:56,353:P4Transfer:ERROR: Replication failure: missing elements in target changelist: E:\p4transfer\ClientSync/ThirdParty/BLAKE3/.gitignore, E:\p4transfer\ClientSync/ThirdParty/BLAKE3/.github/workflows/ci.yml, E:\p4transfer\ClientSync/ThirdParty/BLAKE3/c/.gitignore

This was due to .p4ignore.txt files buried in the depot with the system running p4transfer having P4IGNORE=.p4ignore.txt. The contents of the .p4ignore.txt file were simply:

.git**

Solution was to make sure P4IGNORE was not set on the system. Perhaps p4transfer can unset it somehow when running, or at least prompt the user that it should be unset?

We are seeing below issue with some changes during the migration. We can proceed with the migration by excluding the paths n yaml file but this is the third time we got this error. Any idea what is this about and how to resolve this?

root@dc2-p4-gl-05:/scm/p4transfer # tail -20 log-P4Transfer-20210510090851.log

'rev': 1,
'time': datetime.datetime(2017, 7, 25, 7, 14, 59),
'type': 'text+kx',
'user': 'amirl'}]}]
2021-05-10 09:16:41,946:P4Transfer:DEBUG: src('sync', '//hmallesh_test_transfer/...@=219564')
2021-05-10 09:16:52,093:P4Transfer:DEBUG: src[]
2021-05-10 09:16:53,552:P4Transfer:ERROR: 'utf-8' codec can't decode byte 0xd1 in position 0: invalid continuation byte
Traceback (most recent call last):
File "P4Transfer.py", line 2149, in replicate
num_changes = self.replicate_changes()
File "P4Transfer.py", line 1965, in replicate_changes
fileRevs, branchRevs = self.source.getChange(change['change'])
File "P4Transfer.py", line 1114, in getChange
chRev.updateDigest()
File "P4Transfer.py", line 524, in updateDigest
self.fileSize, self.digest = getKTextDigest(self.fixedLocalFile)
File "P4Transfer.py", line 425, in getKTextDigest
contents = contents.decode()
UnicodeDecodeError: 'utf-8' codec can't decode byte 0xd1 in position 0: invalid continuation byte
2021-05-10 09:16:53,552:P4Transfer:INFO: Sleeping on error for 60 minutes
root@dc2-p4-gl-05:/scm/p4transfer #

It appears a fair bit of work has gone into P4Transfer since April 2nd, which is when the last release was done and tagged. Assuming it's safe, can we please get a new version released.

I'm hitting a lot of errors like

//Foo/Foo/Bar.cpp - moved from //Foo/Bar.cpp but has no matching resolve record; must 'add -d' or 'move' to correct.

The common case in all of them is that the files are moved, edited and integrated in other branches prior to being merged into the current branch as a combined move / edit. This seems to really confuse things. My bat file solution for each file is to:

:: 1. Revert the offending file which will delete it and revert the moved file, bringing it back
p4 -p %target_port% -u %target_user% -c %target_client% revert %moved_file%

:: 2. Edit the original file
p4 -p %target_port% -u %target_user% -c %target_client% edit %deleted_file%

:: 3. Move the original file
p4 -p %target_port% -u %target_user% -c %target_client% move %deleted_file% %moved_file%

:: 4. Force sync the new file
p4 -p %source_port% -u %source_user% -c %source_client% sync -f %moved_file%@%source_change%

I'm pretty sure this is not the correct fix, since it loses the integration history in the present changelist - but I haven't found another way around these errors and they are quite numerous in some cases.

Here's a screenshot of a problematic "move / edit" (highlighted). In this case the file was populated and edited in the working branch (bottom yellow revision) moved in another branch and edited in various other branches, eventually merged back to the working branch new location (top yellow line) as one operation.

Any ideas?

Windows 10
Python 2.7 (was recommended for Unicode compatibility in the documentation)

Issue: p4 reopen will fail if special characters are present in the file path (when fixing file types)
Reason: In fixFileTypes(self, filerevs, openedFiles) the 'fixed' local path is used for reopening: self.p4cmd('reopen', '-t', chRev.type, chRev.fixedLocalFile)

There is probably an inconsistency when to use the 'fixedLocalFile'. If invoking p4cmd one (maybe) only needs to restore the os separator but leave the special characters in their ASCII expanded form? I am currently using chRev.localFile for the reopen in fixFileTypes() and the transfer is proceeding.

log tail:

E:\P4TransferRoot\depot\Projects\MobileGame\Unity5\Assets\Custom\Assets@1x\Textures\Scenery\BabyRoomBed.jpg
2021-02-28 13:38:32,483:P4Transfer:DEBUG: targ('reopen', '-t', 'ubinary', 'E:\P4TransferRoot\depot\Projects\MobileGame\Unity5\Assets\Custom\Assets@1x\Textures\Scenery\BabyRoomBed.jpg')
2021-02-28 13:38:32,483:P4Transfer:ERROR: [P4#run] Errors during command execution( "p4 reopen -t ubinary E:\P4TransferRoot\depot\Projects\MobileGame\Unity5\Assets\Custom\Assets@1x\Textures\Scenery\BabyRoomBed.jpg" )

[Error]: "Invalid changelist/client/label/date '@1x\\Textures\\Scenery\\BabyRoomBed.jpg'."

Traceback (most recent call last):
File "P4TransferDebug.py", line 1823, in replicate
num_changes = self.replicate_changes()
File "P4TransferDebug.py", line 1677, in replicate_changes
targetChange = self.target.replicateChange(filerevs, change, self.source.p4.port)
File "P4TransferDebug.py", line 1027, in replicateChange
self.fixFileTypes(filerevs, openedFiles)
File "P4TransferDebug.py", line 1014, in fixFileTypes
self.p4cmd('reopen', '-t', chRev.type, chRev.fixedLocalFile)
File "P4TransferDebug.py", line 741, in p4cmd
output = self.p4.run(args, **kwargs)
File "C:\Python27\lib\site-packages\P4.py", line 611, in run
raise e
P4Exception: [P4#run] Errors during command execution( "p4 reopen -t ubinary E:\P4TransferRoot\depot\Projects\MobileGame\Unity5\Assets\Custom\Assets@1x\Textures\Scenery\BabyRoomBed.jpg" )

[Error]: "Invalid changelist/client/label/date '@1x\\Textures\\Scenery\\BabyRoomBed.jpg'."

Windows 10
Python 2.7

Some of the folders in our depot have been obliterated. "P4 submit" yields an error as listed below. P4Transfer should skip the 'empty' changelist and continue.

log tail:

2021-02-28 23:00:44,178:P4Transfer:ERROR: [P4#run] Errors during command execution( "p4 submit -i" )

[Error]: 'No files to submit.'

Traceback (most recent call last):
File "P4Transfer.py", line 1817, in replicate
num_changes = self.replicate_changes()
File "P4Transfer.py", line 1671, in replicate_changes
targetChange = self.target.replicateChange(filerevs, change, self.source.p4.port)
File "P4Transfer.py", line 1044, in replicateChange
raise e
P4Exception: [P4#run] Errors during command execution( "p4 submit -i" )

[Error]: 'No files to submit.'

From the p4transfer script, we are getting three sets of notifications

• Regular update , configured with daily frequency
• transfer summary report, configured with weekly frequency
• errors, configured with a 15 minutes frequency

The errors notifications work pretty well, while I have questions about the other two.

The regular update email body sends something like the below, 50 times:
2021-07-14 08:51:49,787:P4Transfer:INFO: Transferring 0 changes
2021-07-14 08:52:49,829:P4Transfer:INFO: Sleeping for 1 minutes
(...)
2021-07-14 09:15:56,121:P4Transfer:INFO: Transferring 0 changes
2021-07-14 09:16:56,164:P4Transfer:INFO: Sleeping for 1 minutes

which seem to be last 50 log instances until the time of the mail is sent (9:16AM in this case)
Would it be possible to get only the log lines of the last 24 hours ( or matching the email frequency) which have only significant values.
In any case, what is the configuration that would switch off the regular update email ?

The transfer summary report email body, received on Monday 12th, has this heading:

Changes transferred since 2021/07/12:09:15:17

It includes the size of transfers since 2021/04/14 , until 2021/07/09
It seems that it always reports from the same date the transfer was started, until the last recorded transfer before the email is sent.
The email body is then going to grow indefinitely.
Also, the heading is giving an incorrect information.

I would like to configure the Summary report to send information covering the time frame of the last week (or the configured frequency of the summary report email), and adjust the heading of the email body to match the time frame.

Thanks

I've been slowly attempting to understand the current state of the repo, but as a new P4Transfer user getting stuck with a simple local test case (Windows 10, Python 3.10), I decided to run the test suite and found a number of errors:

Checkout out tag v1.1.1 however, resulted in a mostly passing test suite with the exception of testUTF16Unsyncable. Are there logs or errors that would be helpful in diagnosing the issue? I apologize in advance if this isn't a helpful report, but I opted to create an issue for awareness since it seems the current automated testing is restricted to Linux.

We have hit cases where files were moved within the current stream but also edited (seemingly to avoid compilation errors since the files are now in a new location). The steps to replicate this behavior seem to be:

1. Open original file for Edit
2. Move (Mark original file for delete)
3. Make changes to the file (which is now Add / Edit)
4. Submit

In replicating this change, p4transfer asserts:

2021-11-05 08:39:34,908:P4Transfer:ERROR:
Traceback (most recent call last):
  File "P4Transfer.py", line 2168, in replicate
    num_changes = self.replicate_changes()
  File "P4Transfer.py", line 1985, in replicate_changes
    targetChange = self.target.replicateChange(fileRevs, srcFileLogs, change, self.source.p4.port)
  File "P4Transfer.py", line 1234, in replicateChange
    self.processChangeRevs(fileRevs, srcFileLogs)
  File "P4Transfer.py", line 1189, in processChangeRevs
    self.replicateIntegration(f)
  File "P4Transfer.py", line 1636, in replicateIntegration
    assert(afterAdd)
AssertionError

I want to transfer a stream from a remote Perforce server that I can't create virtual streams on (I don't have permissions to create one). I tried to use the 'ignore_files' list to filter out folders that I wanted to ignore, like so:

ignore_files:
- "/FolderOne/.*$"
- "/FolderTwo/.*$"
- "/FoderThree/Stuff/.*$

This fails with "Replication failure: missing elements in target changelist:" when validateSubmittedChange calls ChangelistComparer. The diff always contained files causing it to return false. The original code was this:

srcfiles = set([chRev.localFile for chRev in srclist if chRev.localFile not in filesToIgnore])
targfiles = set([chRev.localFile for chRev in targlist])

...but it seems like both the source and target lists of files should filter out the files being ignored. This seemed to work properly:

srcfiles = set([chRev.localFile for chRev in srclist if chRev.localFile not in filesToIgnore])
targfiles = set([chRev.localFile for chRev in targlist if chRev.localFile not in filesToIgnore])

...and the diff was empty with the 'ignore_files' being excluded and not submitted to the local Perforce server.