Using Python 2to3 to Easily Upgrade (Finally)

I finally upgraded one my my repositories using Python 2to3, and I wanted to share how.

Python 2to3 - Introduction

First, if you were not aware, Python 2 is officially end-of-life.

While I am a little late to the update party, I finally got it done on my larger repository.

There are still repositories and libraries out there that still run on Python 2.7, but hopefully they will all phase out soon.

Note that you should already have the tool installed, as Python released it back in 2018.

Usage

Usage of the tool is simple, and you can see all the flags below.

root@kali:~/tools/SecurityTools$ 2to3- --help
Usage: 2to3 [options] file|dir ...

Options:
  -h, --help            show this help message and exit
  -d, --doctests_only   Fix up doctests only
  -f FIX, --fix=FIX     Each FIX specifies a transformation; default: all
  -j PROCESSES, --processes=PROCESSES
                        Run 2to3 concurrently
  -x NOFIX, --nofix=NOFIX
                        Prevent a transformation from being run
  -l, --list-fixes      List available transformations
  -p, --print-function  Modify the grammar so that print() is a function
  -v, --verbose         More verbose logging
  --no-diffs            Don't show diffs of the refactoring
  -w, --write           Write back modified files
  -n, --nobackups       Don't write backups for modified files
  -o OUTPUT_DIR, --output-dir=OUTPUT_DIR
                        Put output files in this directory instead of
                        overwriting the input files.  Requires -n.
  -W, --write-unchanged-files
                        Also write files even if no changes were required
                        (useful with --output-dir); implies -w.
  --add-suffix=ADD_SUFFIX
                        Append this string to all output filenames. Requires
                        -n if non-empty.  ex: --add-suffix='3' will generate
                        .py3 files.

I prefer to run the program recursively, and not automatically write any of the changes.

That said, the -w flag is great if you have a much larger code base with good backups!

Performing My Updates

I wanted to update my SecurityTools repository, as this is where most of my current code lives.

First, I ran the tool on my entire SecurityTools repository. I wrote most of my tools in Python, so I figured that there would be plenty of changes to make.

As you can see, most of the changes were related to the Python 3 print function.

root@kali:~/tools/SecurityTools$ 2to3- .
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Refactored ./AlexaPortScanner/alexaPortScanner.py
--- ./AlexaPortScanner/alexaPortScanner.py	(original)
+++ ./AlexaPortScanner/alexaPortScanner.py	(refactored)
@@ -1,4 +1,4 @@
-from __future__ import print_function
+
 import socket
 import sys
 
RefactoringTool: Refactored ./BinToHex/binToHex.py
--- ./BinToHex/binToHex.py	(original)
+++ ./BinToHex/binToHex.py	(refactored)
@@ -14,4 +14,4 @@
 			ctr = 0
 		ctr += 1
 	shellcode += "\""
-	print shellcode
+	print(shellcode)
RefactoringTool: Refactored ./BurpVERBalyzer/VERBalyzer.py
--- ./BurpVERBalyzer/VERBalyzer.py	(original)
+++ ./BurpVERBalyzer/VERBalyzer.py	(refactored)
@@ -25,7 +25,7 @@
     from org.python.core.util import StringUtil
     import string
 except ImportError:
-    print "Failed to load dependencies."
+    print("Failed to load dependencies.")
 
 VERSION = "1.0"
 callbacks = None
@@ -75,7 +75,7 @@
         callbacks.registerScannerInsertionPointProvider(self)
         callbacks.registerScannerCheck(self)
 
-        print "Successfully loaded VERBalyzer v" + VERSION
+        print("Successfully loaded VERBalyzer v" + VERSION)
         return
 
     # helper method to search a response for occurrences of a literal match string

... <snip> ...

RefactoringTool: Files that need to be modified:
RefactoringTool: ./AlexaPortScanner/alexaPortScanner.py
RefactoringTool: ./BinToHex/binToHex.py
RefactoringTool: ./BurpVERBalyzer/VERBalyzer.py
RefactoringTool: ./DNSRickroll/dnsRickroll.py
RefactoringTool: ./ECBPlaintextAttack/ecbAttack.py
RefactoringTool: ./ECBPlaintextAttack/ecbServer.py
RefactoringTool: ./FileIntegrity/fileintegrity.py
RefactoringTool: ./IpExpander/ipExpander.py
RefactoringTool: ./PortScanner/portScanner.py
RefactoringTool: ./PyDHCPDiscover/dhcpdiscover.py
RefactoringTool: ./PythonShellcode/pythonShellcode.py
RefactoringTool: ./RSAGenKey/genKey.py
RefactoringTool: ./ReverseShell/reverseShell.py
RefactoringTool: ./ZipCracker/zipCracker.py

The only file that had other changes was my old file integrity checker.

root@kali:~/tools/SecurityTools$ 2to3- ./FileIntegrity/fileintegrity.py
RefactoringTool: Skipping implicit fixer: buffer
RefactoringTool: Skipping implicit fixer: idioms
RefactoringTool: Skipping implicit fixer: set_literal
RefactoringTool: Skipping implicit fixer: ws_comma
RefactoringTool: Refactored ./FileIntegrity/fileintegrity.py
--- ./FileIntegrity/fileintegrity.py	(original)
+++ ./FileIntegrity/fileintegrity.py	(refactored)
@@ -6,7 +6,7 @@
 import requests
 import shutil
 import smtplib
-from urlparse import urlparse
+from urllib.parse import urlparse
 
 def calculateOriginalValues(fileUrl, tempFile):
     r = requests.get(fileUrl)
@@ -21,10 +21,10 @@
         calculated_md5 = hashlib.md5(f.read()).hexdigest()
 
     if originalHash == calculated_md5:
-        print "MD5 verified."
+        print("MD5 verified.")
         return None
     else:
-        print "MD5 verification failed!"
+        print("MD5 verification failed!")
         
         file1 = open(fileName, "rb").readlines()
         file2 = open(originalFile, "rb").readlines()
@@ -44,7 +44,7 @@
         
         try:
             while 1:
-                diff_string = diff.next()
+                diff_string = next(diff)
                 if not (diff_string[0] == " "):
                     differences += diff_string + "\r\n"
             if diff is None:
@@ -55,7 +55,7 @@
 
 def sendEmail(configFile, diffString, isSSL, hasAuth):
     config = {}
-    execfile(configFile, config) 
+    exec(compile(open(configFile).read(), configFile, 'exec'), config) 
 
     sender = config["sender"]
     recipient = config["recipient"]
RefactoringTool: Files that need to be modified:
RefactoringTool: ./FileIntegrity/fileintegrity.py

Other than iterators and print methods, I had to learn how to update my execfile call.

After looking at the official documentation, I updated my code to reflect the 2to3 suggestions exactly.

Committing the Changes

When I finished my changes, I went to commit the changes to my repository.

Unfortunately, I was getting an authentication error when I tried to use the git command line tool.

root@kali:~/tools/SecurityTools$ git add .
root@kali:~/tools/SecurityTools$ git commit -m "Updated SecurityTools for Python3"
[master b5bdff4] Updated SecurityTools for Python3
 12 files changed, 66 insertions(+), 67 deletions(-)
root@kali:~/tools/SecurityTools$ git push origin master
Username for 'https://github.com': doyler
Password for 'https://doyler@github.com': 
remote: Invalid username or password.
fatal: Authentication failed for 'https://github.com/doyler/SecurityTools.git/'

I rarely used this for GitHub, so I had to poke around to see why it wasn't working.

As it turns out, I needed to use a personal access token, as opposed to my password.

Once I fixed this, I was able to commit my changes just fine!

root@kali:~/tools/SecurityTools$ git push origin master
Username for 'https://github.com': doyler
Password for 'https://doyler@github.com': 
Enumerating objects: 49, done.
Counting objects: 100% (49/49), done.
Delta compression using up to 16 threads
Compressing objects: 100% (25/25), done.
Writing objects: 100% (25/25), 3.02 KiB | 1.51 MiB/s, done.
Total 25 (delta 14), reused 0 (delta 0)
remote: Resolving deltas: 100% (14/14), completed with 14 local objects.
To https://github.com/doyler/SecurityTools.git
   e082b9f..b5bdff4  master -> master

As you can see, the commit went through, and I updated my repository.

Python 2to3 - Commit

Python 2to3 - Conclusion

I should have updated my repositories sooner, but this was something that I was putting off.

There is still some code in other repositories that I will need to update but let me know if you actually use/need any of it.

Finally, let me know if there are any updates that you'd like to see to any of these tools, or new tools in general!

doyler on Githubdoyler on Twitter
doyler
Ray Doyle is an avid pentester/security enthusiast/beer connoisseur who has worked in IT for almost 16 years now. From building machines and the software on them, to breaking into them and tearing it all down; he's done it all. To show for it, he has obtained an OSCE, OSCP, eCPPT, GXPN, eWPT, eWPTX, SLAE, eMAPT, Security+, ICAgile CP, ITIL v3 Foundation, and even a sabermetrics certification!

He currently serves as a Senior Staff Adversarial Engineer for Avalara, and his previous position was a Principal Penetration Testing Consultant for Secureworks.

When he's not figuring out what cert to get next or side project to work on, he enjoys playing video games, traveling, and watching sports.

Common passed on this blog, I made it to a jam.

Leave a Comment

Filed under Security Not Included

Leave a Reply

Your email address will not be published. Required fields are marked *

*

This site uses Akismet to reduce spam. Learn how your comment data is processed.