A look inside the Samsung S7 camera implementation

This post is less of a guide and more of a collection of thoughts. It assumes some Java and camera operation knowledge.

My S7 has become my main camera since it handles low-light, uploading, water and time zones very well. It does have a few big weakness though, oversharpening being one of them and over-confidence the other. By over-confidence I mean it will gladly lower the shutter speed to 1/4s, thinking OIS will take care of the shake. It can't.

The Samsung camera app on Android Nougat has shed a lot of features in exchange of user-friendliness. Gone are the sharpness controls, OIS cannot be disabled, RAW mode is available only in the "Pro" camera mode. Quite a lot of artificial limitations which third-party apps don't seem to bring back.

Start

To analyze the application we need to take a look at some binaries: SamsungCamera6.apk, semcamera.jar and seccamera.jar. There are some other binaries but they are written in native C (libcamera_client.so) or even for FPGA (e.g. fimc_is_fw2_2l1.bin).

For taking a look inside the jars I'm using dex-tools 2.1, jd-gui 1.4.0 and Eclipse. The classes.dex file is extracted from each app, sent to dex2jar, imported into jd-gui and saved as a zip collection of .java files, then imported into an Eclipse project. There's probably a better workflow for this.

From those various jar files a structure emerges: SemCamera is the bridge between the native camera binary and Android/Java.
FaceAreaManager seems to take care of face detection, HRMSensorFusion enables use of the HR sensor to take selfies.

The camera modes are defined inside the application, even though they need to be downloaded in order to be used:

Interestingly enough, there are a few camera modes which do not have an equivalent app in the Samsung App Store: Antifog, BurstPanorama, Night / NightScene, ProductSearch, ProLite, RichTone, TagShot.

Antifog might be targeted towards the Asian market, as a mode that reduces smog smear. Night might be a sub-mode for the Auto. ProLite might be a Pro implementation for phones with less features (Samsung A7). RichTone might be a sub-mode for Auto+HDR.


I find it both curious and disturbing that features are being hidden from the user at the cost of a "purchase". Not sure if the idea was to monetize camera modes or to simplify camera usage. A simple checkbox list would've sufficed.

Modes

I will now take a look at the "Pro" mode, since it seems to expose the most of the camera functionality.

Features

The menu options are exposed through an implementation of the ShootingModeFeature interface. This tells the framework which options can be changed when the user clicks the 'settings' icon.

In this case, ProFeature sets continous-AF, divideAFAE(?), external memory support, object tracking support, touch AE, touch AF and zoom support to true, and mostly everything else to null or false.
Getting side-tracked here but isLowLightDetectionSupported() will just disable burst mode and add an icon on the screen in the other modes. So it's probably meant as a user-trackable attribute.

Touch EV is supported only inside 'Auto' mode and it seems to add the exposure adjustment slider. Audio focus handling has to do with the shutter sound. [Overlay] effects are available in supported modes upon swiping left.

Some features seem to be disabled because of hardware limitations while others are just to clean up the UI. Unless you test everything yourself, there's no way of knowing whether Super-Resolution-Zoom is supported or not. It probably is.

Overlay Layouts

Snapping the following screenshot took me several minutes with a Ricoh GR, making me appreciate the focus and macro capabilities of the S7.

The Pro mode features an overlay that displays the AF points, like a DSLR. After a bit of navigating around the files, the GL feature seems to be implemented inside MultiAFView:

The class extends GLView[Group] and listens on onMultiAFChanged event. This is triggered by the 'Camera' class' onMultiAutoFocus native event and forwarded to the OverlayLayout controller/listener.
OverlayLayout seems to add overlay hints regarding focus mode (PAF, face, multi-AF) and position, camera mode (Sports, Auto, ...) and the EV-adjustment slider.

In a nutshell, the app receives the focus points from the native application and draws green squares on top of the captured image. All this behavior seems tightly coupled, as though the ASIC/FPGA, the native and the Java app were tightly coordinated. Good for performance, bad for modularity and testing.

As a side note, the native event handling (JNI) is done in a huge if-then-else statement inside SemCamera handleMessage(). 1000+ lines of if statements.


Back to camera modes: most shooting modes are required to handle device orientation by animating the little icons. You saw an image above with the landscape orientation, the portrait one looks like this:

The modes also handle which features and icons are available, upon initialization. They might disable OIS or burst, or enable RAW. They also handle recording video by changing the button layout, animating the rec/pause/play button and showing elapsed or remaining time.

Also visible in the screenshots above are the text renderings for the various settings. I'm pretty sure they are not publicly documented, but the 'A' letter in front of a setting means it's automatically controlled. so A 800 and A 1/10 means that ISO is automatically calculated to be 800 and the shutter speed is calculated to 0.1s.

Exposure calculation

The Pro mode adds itself as a listener to camera sensor data, which forwards sensor telemetrics data: driverResolution, exposureTime, exposureValue, iso, lensPositionCurrent...
When automatic shutter or ISO is enabled, the class iterates through all user-available values and creates a text label. So, instead of shimmering between ISO 613 and ISO 635 the user sees "A 640".

For reference, the available ISO values are: auto, 50, 80, 100, 125, 160, 200, 250, 320, 400, 500, 640, 800. The shutter delays, in microseconds are: auto, 42, 63, 83, 125, 167, 250, 333, 500, 667, 1000, 1333, 2000, 2857, 4000, 5556, 8000, 11111, 16667, 20000, 22222, 33333, 50000, 66667, 100000, 125000, 166667, 250000, 300000, 500000, 1000000, 2000000, 4000000, 8000000, 10000000.
For example, the last shutter delay represents a shutter speed of 10 seconds. So, if you override CameraParameters.getShutterSpeedString(), you can add supplemental values, since the text representation seems to be used for calculating real data as well.

Image adjustments

The Pro mode allows setting a limited range of image parameters: temperature, contrast, shadow/highlight, tint and saturation. Sadly, sharpness is not supported.

The only notable thing in here is the way the parameters are being compiled to be sent to the camera engine - as a plain string:

This looks interesting, it seems the camera engine can take some string modifiers as commands. This is pretty standard for OVF-style mobile cameras, but kind of unexpected in a high-end one.

So there's a nice avenue to explore undocumented camera features.

It seems the sensor (imaging chip) supports a 'FOOD' command. Seems we are getting closer to an answer to "does it make fries as well?".

I haven't had time to map all the exposed features but we do seem to have:

• FOOD=x, LE=y, TO=z, RI=v, B0=w - where x is 0/1, y,z,v and w are rectangular coordinates
• ENHANCED=x (0 or 1)
• FOCUS=x
• customcolor,TE=x,TI=y,CO=z,SA=v,HL=w,SL=z ->temperature, tint, contrast, saturation, highlight, shadows

Camera engine control strings

This is pretty much standardized across the Android ecosystem, but each manufacturer may choose to add its own features.

The "cameraEngine.mParameters.set()" convention seems to be pretty common, but the key-value pairs can get specific.

For Samsung, this seems to be implemented inside SemCamera.Parameters. I'm pasting the semi-complete key list here, but for the remaining values you would have to decompile the app yourself.

ANTIBANDING_50HZ = "50hz";
ANTIBANDING_60HZ = "60hz";
ANTIBANDING_AUTO = "auto";
ANTIBANDING_OFF = "off";
EFFECT_AQUA = "aqua";
EFFECT_BLACKBOARD = "blackboard";
EFFECT_MONO = "mono";
EFFECT_NEGATIVE = "negative";
EFFECT_NONE = "none";
EFFECT_POSTERIZE = "posterize";
EFFECT_SEPIA = "sepia";
EFFECT_SOLARIZE = "solarize";
EFFECT_WHITEBOARD = "whiteboard";
FALSE = "false";
FLASH_MODE_AUTO = "auto";
FLASH_MODE_OFF = "off";
FLASH_MODE_ON = "on";
FLASH_MODE_RED_EYE = "red-eye";
FLASH_MODE_TORCH = "torch";
int FOCUS_DISTANCE_FAR_INDEX = 2;
int FOCUS_DISTANCE_NEAR_INDEX = 0;
int FOCUS_DISTANCE_OPTIMAL_INDEX = 1;
FOCUS_MODE_AUTO = "auto";
FOCUS_MODE_CONTINUOUS_PICTURE = "continuous-picture";
FOCUS_MODE_CONTINUOUS_VIDEO = "continuous-video";
FOCUS_MODE_EDOF = "edof";
FOCUS_MODE_FIXED = "fixed";
FOCUS_MODE_INFINITY = "infinity";
FOCUS_MODE_MACRO = "macro";
KEY_ANTIBANDING = "antibanding";
KEY_AUTO_EXPOSURE_LOCK = "auto-exposure-lock";
KEY_AUTO_EXPOSURE_LOCK_SUPPORTED = "auto-exposure-lock-supported";
KEY_AUTO_WHITEBALANCE_LOCK = "auto-whitebalance-lock";
KEY_AUTO_WHITEBALANCE_LOCK_SUPPORTED = "auto-whitebalance-lock-supported";
KEY_CITYID = "contextualtag-cityid";
KEY_EFFECT = "effect";
KEY_EXPOSURE_COMPENSATION = "exposure-compensation";
KEY_EXPOSURE_COMPENSATION_STEP = "exposure-compensation-step";
KEY_FLASH_MODE = "flash-mode";
KEY_FOCAL_LENGTH = "focal-length";
KEY_FOCUS_AREAS = "focus-areas";
KEY_FOCUS_DISTANCES = "focus-distances";
KEY_FOCUS_MODE = "focus-mode";
KEY_GPS_ALTITUDE = "gps-altitude";
KEY_GPS_LATITUDE = "gps-latitude";
KEY_GPS_LONGITUDE = "gps-longitude";
KEY_GPS_PROCESSING_METHOD = "gps-processing-method";
KEY_GPS_TIMESTAMP = "gps-timestamp";
KEY_HORIZONTAL_VIEW_ANGLE = "horizontal-view-angle";
KEY_JPEG_QUALITY = "jpeg-quality";
KEY_JPEG_THUMBNAIL_HEIGHT = "jpeg-thumbnail-height";
KEY_JPEG_THUMBNAIL_QUALITY = "jpeg-thumbnail-quality";
KEY_JPEG_THUMBNAIL_SIZE = "jpeg-thumbnail-size";
KEY_JPEG_THUMBNAIL_WIDTH = "jpeg-thumbnail-width";
KEY_MAX_EXPOSURE_COMPENSATION = "max-exposure-compensation";
KEY_MAX_NUM_DETECTED_FACES_HW = "max-num-detected-faces-hw";
KEY_MAX_NUM_DETECTED_FACES_SW = "max-num-detected-faces-sw";
KEY_MAX_NUM_FOCUS_AREAS = "max-num-focus-areas";
KEY_MAX_NUM_METERING_AREAS = "max-num-metering-areas";
KEY_MAX_ZOOM = "max-zoom";
KEY_METERING_AREAS = "metering-areas";
KEY_MIN_EXPOSURE_COMPENSATION = "min-exposure-compensation";
KEY_PICTURE_FORMAT = "picture-format";
KEY_PICTURE_SIZE = "picture-size";
KEY_PREFERRED_PREVIEW_SIZE_FOR_VIDEO = "preferred-preview-size-for-video";
KEY_PREVIEW_FORMAT = "preview-format";
KEY_PREVIEW_FPS_RANGE = "preview-fps-range";
KEY_PREVIEW_FRAME_RATE = "preview-frame-rate";
KEY_PREVIEW_SIZE = "preview-size";
KEY_RECORDING_HINT = "recording-hint";
KEY_ROTATION = "rotation";
KEY_SCENE_MODE = "scene-mode";
KEY_SMOOTH_ZOOM_SUPPORTED = "smooth-zoom-supported";
KEY_VERTICAL_VIEW_ANGLE = "vertical-view-angle";
KEY_VIDEO_SIZE = "video-size";
KEY_VIDEO_SNAPSHOT_SUPPORTED = "video-snapshot-supported";
KEY_VIDEO_STABILIZATION = "video-stabilization";
KEY_VIDEO_STABILIZATION_SUPPORTED = "video-stabilization-supported";
KEY_WEATHER = "contextualtag-weather";
KEY_WHITE_BALANCE = "whitebalance";
KEY_ZOOM = "zoom";
KEY_ZOOM_RATIOS = "zoom-ratios";
KEY_ZOOM_SUPPORTED = "zoom-supported";
PIXEL_FORMAT_BAYER_RGGB = "bayer-rggb";
PIXEL_FORMAT_JPEG = "jpeg";
PIXEL_FORMAT_RGB565 = "rgb565";
PIXEL_FORMAT_YUV420P = "yuv420p";
PIXEL_FORMAT_YUV420SP = "yuv420sp";
PIXEL_FORMAT_YUV422I = "yuv422i-yuyv";
PIXEL_FORMAT_YUV422SP = "yuv422sp";
int PREVIEW_FPS_MAX_INDEX = 1;
int PREVIEW_FPS_MIN_INDEX = 0;
SCENE_MODE_ACTION = "action";
SCENE_MODE_AUTO = "auto";
SCENE_MODE_BARCODE = "barcode";
SCENE_MODE_BEACH = "beach";
SCENE_MODE_CANDLELIGHT = "candlelight";
SCENE_MODE_FIREWORKS = "fireworks";
SCENE_MODE_HDR = "hdr";
SCENE_MODE_LANDSCAPE = "landscape";
SCENE_MODE_NIGHT = "night";
SCENE_MODE_NIGHT_PORTRAIT = "night-portrait";
SCENE_MODE_PARTY = "party";
SCENE_MODE_PORTRAIT = "portrait";
SCENE_MODE_SNOW = "snow";
SCENE_MODE_SPORTS = "sports";
SCENE_MODE_STEADYPHOTO = "steadyphoto";
SCENE_MODE_SUNSET = "sunset";
SCENE_MODE_THEATRE = "theatre";
SUPPORTED_VALUES_SUFFIX = "-values";
TRUE = "true";
WHITE_BALANCE_AUTO = "auto";
WHITE_BALANCE_CLOUDY_DAYLIGHT = "cloudy-daylight";
WHITE_BALANCE_DAYLIGHT = "daylight";
WHITE_BALANCE_FLUORESCENT = "fluorescent";
WHITE_BALANCE_INCANDESCENT = "incandescent";
WHITE_BALANCE_SHADE = "shade";
WHITE_BALANCE_TWILIGHT = "twilight";
WHITE_BALANCE_WARM_FLUORESCENT = "warm-fluorescent";

Example: to set the focus mode to infinity we need to set KEY_FOCUS_MODE to FOCUS_MODE_INFINITY, or - mParameters.set("focus-mode", "infinity").

That looks fine but it does not seem to be the complete list. I still seem to be missing the sharpness property. Also "ois" is again not listed. A quick search through related files yields some results:

The screenshot above lists all available keys and all of their possible values. Unfortunately it's almost impossible to tell which is which. Since there are only a handful of values, I guess one could try all possible combinations and use some common sense to derive all features.

Inside com.sec.android.camera.util.CameraParameters we get an intersecting list:

"scene-mode"
"flash-mode"
"picture-size"
"focus-mode"
"exposure-compensation"
"effect"
"wb-k"
"whitebalance"
"iso"
"exposure-time"
"jpeg-quality"
"zoom"
"metering"
"rt-hdr"
"video-size"
"jpeg-quality"
"video-stabilization"
"fast-fps-mode"
"motion-speed"
"camera_id"
"shot-mode"
"focus-distance"
"picture-format"
"intensity"
"vignette"
"wb-level"
"multi-af"

There's not much extract from those and certainly no sharpness setting.

The problem seems to be that the parameters are dynamically available (determined) at runtime, which is a bit of a bummer, but nothing Google can't solve. After some searching, it seems the relevant terms are "min-sharpness max-sharpness" which will lead you to a list of sites . Of those, a few list some manufacturer-specific keys:

http://lackeysoft.com/forums/viewtopic.php?f=6&t=21

https://gist.github.com/anthonykeane/5963414 

https://github.com/arieljannai/learning-things/blob/master/android-camera-parameters.txt

https://stackoverflow.com/questions/45501223/android-camera-set-metering-spot-position

Interesting settings

We do see a list of interesting strings, some of them exposed just summarily, others not at all:

• smilshot - the smile trigger
• object-tracking and face-detection
• high-dynamic-range
• ois - so far 'center' and 'still' are used
• light-fx
• scene-mode -with steadyphoto and others as value
• red-eye torch fillin slow red-eye-fix - flash_mode values
• off still video zoom sine_x sine_y center - possibly "ois" values
• sw-vdis - video stabilization, "on" or "off"
• focus-mode: auto, infinity, fixed, manual, face-priority, continuous-picture, continuous-video, object-tracking-picture, object-tracking-video, macro, edof, multi, touch, self
• horizontal-view-angle, vertical-view-angle

There are many others, some of them are likely to be only listener events (like "low-light"), some of them are possibly hardware-dependent. But there is still a lot of room to explore.

Sports mode

For completeness, a quick analysis of the 'Sports' camera add-on. This mode is interesting as it seems to be tailored for the best camera performance and it has one of the lowest line code counts. In fact it's so small I can just paste it here.

The ShootingModeFeature implementation enables C-AF, SD-Card saving, object tracking, touch-AF and zoom, while disabling everything else. It disables low-light detection, gesture control, face detection, effects and AF/AE lock. 

public class Sports
  implements ShootingMode
{
  private static final String TAG = "Sports";
  private BaseMenuController mBaseMenuController = null;
  private CameraContext mCameraContext = null;
  private Engine mEngine = null;
  private MenuManager mMenuManager = null;
  private TimerCountingMenu mTimerMenu = null;

  public Sports(CameraContext paramCameraContext, CameraSettings paramCameraSettings)
  {
    this.mCameraContext = paramCameraContext;
  }

  public boolean isCapturing()
  {
    return false;
  }

  public boolean isChangingShootingModeDisabled()
  {
    return false;
  }

  public boolean isZoomDisabled()
  {
    return false;
  }

  public void onActivate(Engine paramEngine)
  {
    this.mEngine = paramEngine;
    this.mBaseMenuController.setSideQuickSettingItems(new int[0]);
    this.mBaseMenuController.showView(-1);
  }

  public boolean onActivityTouchEvent(MotionEvent paramMotionEvent)
  {
    return false;
  }

  public void onChangeShootingModeParameters(SemCamera.Parameters paramParameters)
  {
    paramParameters.set(CameraParameters.getModeString(2), CameraParameters.getSceneModeString(2));
  }

  public void onCreateView(GLContext paramGLContext, GLViewGroup paramGLViewGroup1, GLViewGroup paramGLViewGroup2, BaseMenuController paramBaseMenuController, MenuManager paramMenuManager)
  {
    SemLog.secD("Sports", "onCreateView");
    this.mBaseMenuController = paramBaseMenuController;
    this.mMenuManager = paramMenuManager;
  }

  public boolean onImageStoringCompleted()
  {
    return false;
  }

  public void onInactivate()
  {
    SemLog.secD("Sports", "onInactivate");
    this.mBaseMenuController.removeSideQuickSettingItems();
  }

  public boolean onKeyDown(int paramInt, KeyEvent paramKeyEvent)
  {
    return false;
  }

  public boolean onKeyUp(int paramInt, KeyEvent paramKeyEvent)
  {
    return false;
  }

  public boolean onMenuSelected(int paramInt1, int paramInt2)
  {
    return false;
  }

  public boolean onRecordKeyReleased()
  {
    this.mCameraContext.switchToRecordingMode();
    return false;
  }

  public boolean onShutterKeyLongPressed()
  {
    return false;
  }

  public boolean onShutterKeyPressed()
  {
    return false;
  }

  public boolean onShutterKeyReleased(int paramInt)
  {
    this.mEngine.handleSingleShutterReleased(paramInt);
    return false;
  }

  public void onSingleCaptureEvent(int paramInt)
  {
    switch (paramInt)
    {
    }
    for (;;)
    {
      return;
      if (this.mEngine.getIntervalCaptureCount() != 0)
      {
        this.mEngine.cancelIntervalCapture();
        this.mBaseMenuController.enableView(192);
        this.mBaseMenuController.showView(-1);
      }
    }
  }

  public void onTimerEvent(int paramInt)
  {
    SemLog.secI("Sports", "onTimerEvent: " + paramInt);
    if (!this.mMenuManager.isActive(61)) {
      this.mTimerMenu = ((TimerCountingMenu)this.mMenuManager.showMenu(61));
    }
    if (this.mTimerMenu != null) {
      this.mTimerMenu.updateTime(paramInt);
    }
    if (paramInt == 0)
    {
      if (!this.mCameraContext.isCapturing()) {
        break label136;
      }
      if ((this.mEngine.getIntervalCaptureCount() == 0) || (this.mEngine.getIntervalCaptureCount() >= 3))
      {
        this.mBaseMenuController.showView(-1);
        this.mBaseMenuController.enableView(704);
      }
    }
    for (;;)
    {
      return;
      label136:
      this.mBaseMenuController.showView(384);
      this.mBaseMenuController.enableView(128);
    }
  }

  public boolean onVideoStoringCompleted()
  {
    return false;
  }
}

There's really one single active line in the entire listing: paramParameters.set("scene-mode", "sports"); [edited for brevity]

As far as I can see, the sports mode disables all features and also initializes the sensor into "Sports" mode. It's interesting to figure out why Samsung thought AE/AF lock has an effect on performance, it might have something to do with the way Sony/Isocell has implemented the sensor.

Other notes

There are two image sensor implementations on the S7, revealed by the firmware files.

One of them is the Isocell S5K2L1 and the other one is Sony IMX260, I have the ISOCELL one. According to a respected site there is little difference between them. My amateur photographer's eye can hardly distinguish any difference, so they are probably equivalent for everyday use. However I don't know if the implemented features differ in any way.

Conclusion

Well, it was a long post with no real conclusion. But I hope you learned a bit about the inner workings of a Samsung phone camera and how it ties to the Android UI.

I haven't delved into a lot of the implementation details, but there are some interesting development options, though certainly not cutting-edge.

For you, the reader, there are a lot of options and undocumented settings to explore. Since the post is analysing an Androind N[ougat] implementation, things are subject to change. 

My plan is to inject some customizations into the Camera Api/App, assuming I still have the phone by the time I'm done with them. Hopefully it can be done without the Xposed framework.

I'm looking for a way to override the ISO/exposure computation in at least one camera mode and lower the sharpness level. This seems doable so far. Exposing extra features would be real nice. A model for the level of exposure is CHDK.

Whether this helps you develop a custom Samsung camera app or lets you improve your existing one, I would appreciate any kind of pingback.

Subjectively, the iPhone7 focuses faster and more precisely, has a more accurate white balance and has a higher percentage to get the right shot. But if you pamper the S7, you can get shots that are twice as good, assuming you are willing to spend some time for setup.