the great adobe cs2 case-sensitive adventure, part 3: gdb magic

I fiddled around with shark and sampler a bit, to see if they could offer any insights into the differences between a copy of photoshop running on a case-sensitive filesystem vs. a case-insensitive one, but the information reported is too coarse-grained: the tools will extract when open() is being called, but won't tell you anything about the arguments. at least they offer a nice list of low-level file i/o system calls.

next up, gdb. using gdb, load a copy of photoshop:
(gdb) file /Volumes/PSCaseSensitive/Adobe\ Photoshop\ CS2.app
Reading symbols for shared libraries ............................ done
Reading symbols from /Volumes/PSCaseSensitive/Adobe Photoshop CS2.app/Contents/MacOS/Adobe Photoshop CS2...done.
try setting a breakpoint on open(), and do some fancy scripting to only stop when open() fails. I recommend putting the following into a separate file:

set logging file ~/analysis.txt

define mr-analyze
 # set a break in the calling routine - this is a cheap and dirty version of
 # mr, as mr doesn't seem to work all the time here. maybe because not
 # enough preamble has run when the bp is hit? anyway, this works.
 break *$lr
 set $pathname = $r3

 # continue to calling routine
 c
 
 # will break here in calling routine

 # delete this breakpoint
 d $bpnum
 
 # dump this to a file
 set logging on
 
 # print some stuff if open returned -1
 echo
 if $r3==-1
  echo
  info symbol $pc
  printf "ERROR OPENING FILE: %s\n", $pathname
 else
  echo
  info symbol $pc
  printf "fdesc: %d %s\n", $r3, $pathname
 end
 
 set logging off
 
 # continue to next open() break
 c
end

set up the breakpoint on open() or stat() or whatever file i/o call you want to look at. this macro assumes that the filename is being passed in $r3, if not, adjust accordingly.
(gdb) break open
Breakpoint 1 at 0x90001ce0
(gdb) source ~/analyze_open
(gdb) commands $bpnum
Type commands for when breakpoint 1 is hit, one per line.
End with a line saying just "end".
>silent
>mr-analyze
>end
(gdb) run
Starting program: /Volumes/PSCaseSensitive/Adobe Photoshop CS2.app/Contents/MacOS/Adobe Photoshop CS2
Reading symbols for shared libraries .....................................................................+++++++++++++........++++++++ done
Breakpoint 2 at 0x909c1284

Breakpoint 2, 0x909c1284 in secure_open ()
Breakpoint 3 at 0x90b1d880
...
Breakpoint 872, 0x90b3a330 in POSIXGetADInfo ()
POSIXGetADInfo(char const*, stat*, unsigned char, FInfo*, FXInfo*, unsigned long*, unsigned long*) + 96 in section LC_SEGMENT.__TEXT.__text of /System/Library/Frameworks/CoreServices.framework/Versions/A/Frameworks/CarbonCore.framework/Versions/A/CarbonCore
ERROR OPENING FILE: /Network/._Library
Breakpoint 873 at 0x90b156e4
...
and so on. run this again on the case-insensitive install, and diff the two log files. I did this with a stack of different system calls, but didn't come across anything too telling. the most interesting part is that the asn.framework (adobe serial number?) in the case-insensitive run reads the following two files:
fdesc: 13 /Users/thrust/Desktop/Adobe Photoshop CS2.app/Contents/Resources/ASN/PS.SIF
fdesc: 13 /Users/thrust/Desktop/Adobe Photoshop CS2.app/Contents/Resources/ASN/PS.SIF
but the case-sensitive run doesn't even try. shortly after doing so, the license manager successfully runs in the case-insensitive version.