Thursday, November 11, 2010

Simple Effect

Pivot nicely provides a few nice effects for items. This is useful for images, etc but if you want to make a simple text-based header it needs the tiniest bit of tweaking.
<Label text="Your Title" styles="{font:'Arial bold 36',color:'#a0a0a0'}">
<decorators>
<effects:Reflection/>
</decorators>
</Label>
What we need is to translate the reflection so it overlaps the text a little.
<Label text="Your Title" styles="{font:'Arial bold 36',color:'#a0a0a0'}">
<decorators>
<myeffects:ConfigurableReflection yTranslate="-20"/>
</decorators>
</Label>
We can just modify the Reflection effect and two properties for x & y reflection translation and tweak where the reflection is drawn. Here's the modified ConfigurableReflection class
public class ConfigurableReflection extends ReflectionDecorator {

    private Component component = null;
    private Graphics2D graphics = null;
    private BufferedImage componentImage = null;
    private Graphics2D componentImageGraphics = null;

    private int yTranslate = 0;
    private int xTranslate = 0;

    public void setXTranslate(int xTranslate) {
        this.xTranslate = xTranslate;
    }

    public void setYTranslate(int yTranslate) {
        this.yTranslate = yTranslate;
    }

    public int getXTranslate() {
        return xTranslate;
    }

    public int getYTranslate() {
        return yTranslate;
    }

    

    @Override
    public Graphics2D prepare(Component component, Graphics2D graphics) {
        this.component = component;
        this.graphics = graphics;

        int width = component.getWidth();
        int height = component.getHeight();

        componentImage = new BufferedImage(width, height, BufferedImage.TYPE_INT_ARGB);
        componentImageGraphics = componentImage.createGraphics();

        // Clear the image background
        componentImageGraphics.setComposite(AlphaComposite.Clear);
        componentImageGraphics.fillRect(0, 0, componentImage.getWidth(), componentImage.getHeight());

        componentImageGraphics.setComposite(AlphaComposite.SrcOver);

        return componentImageGraphics;
    }

    @Override
    public void update() {
        // Draw the component
        graphics.drawImage(componentImage, 0, 0, null);

        // Draw the reflection
        int width = componentImage.getWidth();
        int height = componentImage.getHeight();

        GradientPaint mask = new GradientPaint(0, height / 4f, new Color(1.0f, 1.0f, 1.0f, 0.0f),
                0, height, new Color(1.0f, 1.0f, 1.0f, 0.5f));
        componentImageGraphics.setPaint(mask);

        componentImageGraphics.setComposite(AlphaComposite.DstIn);
        componentImageGraphics.fillRect(0, 0, width, height);

        componentImageGraphics.dispose();
        componentImageGraphics = null;

        componentImage.flush();

        graphics.transform(getTransform(component));

        graphics.drawImage(componentImage, 0, 0, null);

        componentImage = null;
        component = null;
        graphics = null;
    }

    @Override
    public Bounds getBounds(Component component) {
        // MODIFICATION for new translation
        return new Bounds(0, 0, component.getWidth() + xTranslate, component.getHeight() * 2 + yTranslate);
    }

    @Override
    public AffineTransform getTransform(Component component) {
        AffineTransform transform = AffineTransform.getScaleInstance(1.0, -1.0);
        // MODIFICATION for new translation
        transform.translate(xTranslate, -((component.getHeight() * 2) + yTranslate));

        return transform;
    }
}

Thursday, September 16, 2010

Simple netbeans Pivot module

I got a little tired re-running my entire app just to see what the current pivot layout would look like, so here's a quick and dirty plugin for Netbeans 6.8 which does the following:
  • Pivot 1.5.1 library
  • XML-editing and formatting for .wtkx files
  • right-click, 'Preview WTKX' function
download To use this module on a pivot project, Right-click on project, select properties and add the new Pivot 1.5.1 library to your project. When viewing a file, you first need to build your project then right click on the .wtkx files to preview.

Monday, August 16, 2010

Extracting selected pivot tree items

The pivot TreeView only returns selected paths. A path is a list of list indexes which refer to the underlying data model of a tree. Unfortunately, the treeview doesn't have a convenience method for extracting the items referenced by those paths. Here's that missing method:
public static List extractSelectedObjects(TreeView tv)
    {
        List retList = new ArrayList();
        
        for (Path p : tv.getSelectedPaths())
        {
            Object currBranch = tv.getTreeData();

            for (int i : p)
            {
                currBranch = ((List) currBranch).get(i);
            }
            retList.add(currBranch);
        }

        return retList;
    }

Thursday, August 12, 2010

Preventing grizzly timeouts

In grizzly, the default timeout for an http process is 5 minutes. This applies to the total activity time. In most cases, this is fine, however for long uploads you may need to extend this. Oh say for uploading 50g files. There is a switch for disabling timeouts, however this is global. What we need is a timeout for connections that are still active. Inactive connections should close 5m after last activity. Grizzly logs the following:
Aug 12, 2010 3:17:25 PM com.sun.grizzly.http.KeepAliveThreadAttachment timedOut
WARNING: GRIZZLY0023: Interrupting idle Thread: Grizzly-8080(2).
Aug 12, 2010 3:17:27 PM com.sun.grizzly.http.KeepAliveThreadAttachment timedOut
WARNING: GRIZZLY0023: Interrupting idle Thread: Grizzly-8080(3).
The somewhat messy solution is the pick apart the thread that your request is running in and toggle timeouts as needed.
while ((read = in.read(byteArray)) != -1)
            {
                if ((System.currentTimeMillis() - lastXmit) > 5000)
                {
                    HttpWorkerThread thread = (HttpWorkerThread)Thread.currentThread();
                    ((SelectionKeyAttachment) thread.getProcessorTask().getSelectionKey().attachment()).setTimeout(lastXmit);
                    lastXmit = System.currentTimeMillis();
                }
                buffer.clear();
                buffer.limit(read);
                openFile.write(buffer);
            }

Tuesday, June 8, 2010

Apache releases Pivot 1.5

The Apache Pivot project released version 1.5. download.

Friday, May 21, 2010

Head bashing

When working with smartclient and trying to get pages to render on IE, you may see a lot of random "'something' is null or not an object" messages. This is caused by a dangling , in your list of items. Most times, look at the members or fields array for the little beastie hanging around at the end.

Tuesday, April 13, 2010

J2Weeeeeeeeeee

As strongbad would say, for good or for awesome...

@Path("/orders/")
@Interceptors(CallAudit.class)
@Stateless public class OrderService {

  @EJB BillingService billing;
  @EJB DeliveryService delivery;
  @EJB Warehouse warehouse;

  @PUT
  @Produces({"application/xml","application/json"})
  @Consumes({"application/xml","application/json"})
  public Order order(Order newOrder){
      Order order = warehouse.checkout(newOrder);
      billing.payForOrder(order);
      delivery.deliver(order);
      return order;
  }

  @GET
  @Path("{orderid}/")
  @Produces({"application/xml","application/json"})
  public Order status(@PathParam("orderid") long orderId){
     return delivery.status(orderId);
  }
}
Full details here

Monday, March 22, 2010

Storage Success

Our web-based storage system was able to completely saturate the gigabit link to a compute cluster. The serving nodes were two Sun x2100's, each with an older Apple XRAID attached. All access was through the grizzly based webserver. Separate testing was able to show this system could handle between 1200 and 2200 small file requests/s per node.

Wednesday, March 10, 2010

Being Evil

Someone was using SiteSucker on a demo site and was behaving badly, this site also hosts our redmine install on the same DB and started to slow down my work. To block we had a few options, iptables the IP, block the IP w/in apache (htaccess) or do the evil thing, just block based on their user-agent. I chose the latter.
RewriteEngine on
...
RewriteCond %{HTTP_USER_AGENT} ^SiteSucker2.3.1
RewriteRule ^(.*) /badbot.html
This is evil since the persons crawler will get out badbot.html page. The person will then use firefox/ie/whatever to browse to the page and look to see why it's not working, but since it's sending a differend user-agent, they will be allowed to browse. For anyone who knows how to configure a crawler it isn't an issue changing the supplied agent, but then again that person will likely be able to control their crawler and not kill my web server. Here's a snapshot of the logs showing the crawler pulling badbot (size 174) followed by a browse attempt from safari.
82.232.60.250 - - [09/Mar/2010:19:52:41 -0500] "GET /cgi-bin/isadg/viewitem.pl?item=14379 HTTP/1.1" 200 174 "http://narademo.umiacs.umd.edu/cgi-bin/isadg/viewseries.pl?seriesid=1054" "SiteSucker2.3.1 CFNetwork/438.14 Darwin/9.8.0 (i386) (MacBook1%2C1)"
82.232.60.250 - - [09/Mar/2010:19:52:42 -0500] "GET /cgi-bin/isadg/viewitem.pl?item=14377 HTTP/1.1" 200 174 "http://narademo.umiacs.umd.edu/cgi-bin/isadg/viewseries.pl?seriesid=1054" "SiteSucker2.3.1 CFNetwork/438.14 Darwin/9.8.0 (i386) (MacBook1%2C1)"
82.232.60.250 - - [10/Mar/2010:11:03:13 -0500] "GET /cgi-bin/isadg/viewrg.pl?rgid=44 HTTP/1.1" 200 2286 "-" "Mozilla/5.0 (Macintosh; U; Intel Mac OS X 10_5_8; fr-fr) AppleWebKit/531.21.8 (KHTML, like Gecko) Version/4.0.4 Safari/531.21.10"

Friday, March 5, 2010

Test details

Here's the details of the performance testing listed below, hardware etc and the raw IOZone numbers. Client Details:
  • Dual-Core AMD Opteron(tm) Processor 1220 (Sun 2100)
  • 2.8Ghz, 2GB Memory, 1Gbps network, RHEL4
Server Details:
  • Intel Xeon 5120, 1.86Ghz (dell 1950)
  • 1GB Memory, 1Gbps network, RHEL5
  • Perc5i raid controller setup in 2 7-disk raid 5 volumes
  • Disk: Dell MD1000 w/ 750Gb SATA 7200rpm drives
Untuned results (4G test, 4k records) in KB/s
                                                   random  random    bkwd      
     KB  reclen   write rewrite    read    reread    read   write    read 
4194304    4096   93137   68662    28869    29300   23118   57440   23186 
Tuned results:
                                                   random  random    bkwd
     KB  reclen   write rewrite    read    reread    read   write    read
4194304    4096   95360   71281   108200   112603   67761   69528   68227
Notice the huge gains in read performance 28MB/s to 108MB/s for sequential and 23MB/s to 67MB/s for non-sequential. Here's the theoretical max based on running iozone on the server. It looks like nfs is adding overhead on rewrites and possibly random reads and writes with sequential reads/writes bottlenecking at the 1Gbps connection
                                                   random  random    bkwd      
     KB  reclen   write rewrite    read    reread    read   write    read 
2097152    4096  302379  234327   174385   196905  102032  144843   99866
  • Raw server performance:
  • Untuned performance: xls txt
  • Tuned performance: xls txt
  • Server to raid xls txt

Thursday, March 4, 2010

NFS Performance

We're testing a moderately fast nfs server. Testing on our data set of large files, ~100M we were seeing performance a little over 100MB/s. Now, when we shared the same filesystem out via nfs performance dropped to a mere 44MB/s and no matter what the r/wsize of the client, would not increase. Both machines are connected via a 1Gbps link that has been shown to move 941mb/s. Here's a snapshot what the server workload looked like on the server. The first line shows the local testing, notice the 108MB/s and the second shows the new access. The interesting part is the device utilization during the tests, even though we were pushing half the requests/second and moving half the bandwidth, we were at 90% with a higher wait time 4ms vs 1.7ms.
Device:         rrqm/s   wrqm/s   r/s   w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util
sdb              30.50     0.00 906.00  0.00 108530.00     0.00   239.58     1.61    1.78   1.08  98.15
sdb             870.00     0.00 500.50  0.00 44626.00     0.00   178.33     2.06    4.09   1.83  91.55
Now, let's tweak the ioscheduling a little by setting slice_idle to 1 as recommended in the dell guide below. 'echo 1 > /sys/block/sdb/queue/iosched/slice_idle' This affects how long the io scheduler will wait in each queue before looking for more work. On local access where you will have lots of low latency requests you probably want to wait for more work, however over a network with lots of clients you will want to switch as soon as possible
sdb            1164.18     0.00 1049.75  0.00 71150.25     0.00   135.56     2.34    2.23   0.87  91.79
let's go a little further and set it to 0
sdb             144.00     0.00 2428.50  0.00 92034.00     0.00    75.79     3.95    1.63   0.33  79.60
Now we're up to 92MB/s which is above 90% of what our network can do. Good enough!

Thursday, February 25, 2010

Misc Pivot stuff

Padding choices
In wtkx, adding padding usually looks like: styles="{padding:10}". Nice if you want 10px all around, but you can also specify each side styles="{padding:{top:1,left:2,bottom:3,right:4}}".

Adding validators inline
Normally to add a validator to a component, you code something similar to
@WTKX
private TextInput maxFileSize;
...
...
maxFileSize.setValidator(new IntRangeValidator(0, Integer.MAX_VALUE));
Seems like a bit of a waste? In WTKX, you can include any object that has a no-argument constructor. So we can inline the whole thing.
  1. Include the package to your object as part of the xml namespace (notice xmlns:text)
  2. Define the validator.
  3. attach it to your component
<Sheet wtkx:id="exportDialog" title="Generate Crawl List"
    xmlns:wtkx="http://pivot.apache.org/wtkx"
    xmlns:text="org.apache.pivot.wtk.text.validation"
    xmlns="org.apache.pivot.wtk"  styles="{padding:10}">
 <wtkx:define>
     <text:intrangevalidator id="numvalidator"
       minimum="0" maximum="2147483647"/>
 </wtkx:define>
 <content>
...
...
   <textinput textkey="maxFileSize" validator="$numvalidator"
    id="maxFileSize" text="0"/>
...
 </content>
</Sheet>

Tuesday, February 23, 2010

SASL Plain Server

For some reason SUN includes support for sasl PLAIN in their SaslClient, but not SaslServer. Here's a quick hack to implement a plain sasl server. As per rfc4616.txt, there is only message, three UTF-8 strings, max length 255 delimited by null (\u0000) sent from the client. There is no server response.
import java.io.IOException;
import java.io.UnsupportedEncodingException;
import javax.security.auth.callback.Callback;
import javax.security.auth.callback.CallbackHandler;
import javax.security.auth.callback.UnsupportedCallbackException;
import javax.security.sasl.AuthorizeCallback;
import javax.security.sasl.SaslException;
import javax.security.sasl.SaslServer;

/**
 * Simple sasl server to decode a
 * @author toaster
 */
public class PlainSaslServer implements SaslServer
{

    public boolean complete = false;
    private String authID = null;
    private CallbackHandler cbh;
    
    PlainSaslServer(CallbackHandler cbh)
    {
        this.cbh = cbh;
    }

    public String getMechanismName()
    {
        return SaslConstants.SASL_SERVICE;
    }

    public byte[] evaluateResponse(byte[] response) throws SaslException
    {
        String authnID = null;
        String authzID = null;
        String password = null;

        // Decode response as per rfc4616.txt
//       The formal grammar for the client message using Augmented BNF [ABNF]
//       follows.
//
//       message   = [authzid] UTF8NUL authcid UTF8NUL passwd
//       authcid   = 1*SAFE ; MUST accept up to 255 octets
//       authzid   = 1*SAFE ; MUST accept up to 255 octets
//       passwd    = 1*SAFE ; MUST accept up to 255 octets
//       UTF8NUL   = %x00 ; UTF-8 encoded NUL character
//
//       SAFE      = UTF1 / UTF2 / UTF3 / UTF4
//                   ;; any UTF-8 encoded Unicode character except NUL
//
//       UTF1      = %x01-7F ;; except NUL
//       UTF2      = %xC2-DF UTF0
//       UTF3      = %xE0 %xA0-BF UTF0 / %xE1-EC 2(UTF0) /
//                   %xED %x80-9F UTF0 / %xEE-EF 2(UTF0)
//       UTF4      = %xF0 %x90-BF 2(UTF0) / %xF1-F3 3(UTF0) /
//                   %xF4 %x80-8F 2(UTF0)
//       UTF0      = %x80-BF

        int start = 0;
        int end = 0;
        int elementIdx = 0;

        try
        {


            for (byte b : response)
            {
                if (b == '\u0000')
                {
                    // empty string, only authzid allows this
                    if (end - start == 0)
                    {
                        if (elementIdx != 0)
                        {
                           throw new SaslException("null auth data");
                        }

                    } // data, wa-hoo
                    else
                    {
                        String element = new String(response, start, end - start, "UTF-8");
                        start = end + 1;

                        switch (elementIdx)
                        {
                            case 0:
                                authzID = element;
                                break;
                            case 1:
                                authnID = element;
                                break;
                            default:
                                throw new SaslException("Unexpected data in packet");
                        }
                    }
                    elementIdx++;

                }
                end++;
            }

            if (start == end)
            {
                throw new SaslException("null auth data");
            }

            password = new String(response, start, end - start, "UTF-8");


        } catch (UnsupportedEncodingException e)
        {
            throw new SaslException("utf-8 encoding");
        }


        ExternalValidationCallback evc = new ExternalValidationCallback(authnID, password);
        AuthorizeCallback ac = new AuthorizeCallback(authnID, authzID);

        Callback[] cbList = new Callback[2];

        cbList[0] = evc;
        cbList[1] = ac;

        try
        {
            if (password == null || authnID == null)
            {
                throw new SaslException("null auth data");
            }

            cbh.handle(cbList);
            if (!evc.isValidated())
            {
                throw new SaslException("cannot validate password");
            }

            if (!ac.isAuthorized())
            {
                throw new SaslException("user not authorized");
            }

            complete = true;
            return null;

        } catch (UnsupportedCallbackException ex)
        {
            throw new SaslException("unsupported callback", ex);

        } catch (IOException e)
        {
            if (e instanceof SaslException)
            {
                throw (SaslException) e;
            }
            throw new SaslException("Callback error", e);
        }

    }

    public boolean isComplete()
    {
        return complete;
    }

    public String getAuthorizationID()
    {
        if (!complete)
        {
            throw new IllegalStateException("not complete");
        }
        return authID;
    }

    public byte[] unwrap(byte[] incoming, int offset, int len)
    {
        if (!complete)
        {
            throw new IllegalStateException("not complete");
        }
        final byte[] result = new byte[len];
        System.arraycopy(incoming, offset, result, 0, len);
        return result;

    }

    public byte[] wrap(byte[] outgoing, int offset, int len)
    {
        if (!complete)
        {
            throw new IllegalStateException("not complete");
        }

        final byte[] result = new byte[len];
        System.arraycopy(outgoing, offset, result, 0, len);
        return result;

    }

    public Object getNegotiatedProperty(String propName)
    {
        return null;
    }

    public void dispose() throws SaslException
    {
    }
}
This also uses the following callback to validate the client's username and password. The SaslServer will throw a SaslException if the isValidated returns false.
public class ExternalValidationCallback implements Callback
{

    private String username;
    private String password;
    private boolean validated = false;

    public ExternalValidationCallback(String username, String password)
    {
        this.username = username;
        this.password = password;
    }

    public String getPassword()
    {
        return password;
    }

    public String getUsername()
    {
        return username;
    }

    public boolean isValidated()
    {
        return validated;
    }

    public void setValidated(boolean validated)
    {
        this.validated = validated;
    }
}
Now after we have the new callback and server we need to register it as a provider w/in java. This requires that we create a server factory and simple security provider to register the factory. The server factory will need to check the sasl property POLICY_NOPLAINTEXT to see if its allowed to return the PLAIN server.
public class PlainSaslServerFactory implements SaslServerFactory
{

    private static String[] mechanisms =
    {
        "PLAIN"
    };

    public SaslServer createSaslServer(String mechanism, String protocol,
            String serverName,
            Map props, CallbackHandler cbh) throws SaslException
    {

        if (!mechanisms[0].equals(mechanism) || cbh == null)
            return null;
       return new PlainSaslServer(cbh);

    }

    public String[] getMechanismNames(Map props)
    {
        if ("true".equals(props.get(Sasl.POLICY_NOPLAINTEXT)))
            return new String[0];
        return mechanisms;
    }
}
Now, create the java security provider.
public final class MySecurityProvider extends Provider
{

    public MySecurityProvider()
    {
        super("My Provider", 1.0 , "Simple sasl plain provider");
        put("SaslServerFactory.PLAIN", "org.swap.provider.PlainSaslServerFactory");
    }

}
Putting it all together
// register provider
Security.addProvider(new MySecurityProvider());
// create callback
CallbackHandler myHandler = new CallbackHandler() {
public void handle(Callback[] callbacks) throws IOException, UnsupportedCallbackException
    {
 for (Callback cb : callbacks)
        {
            if (cb instanceof ExternalValidationCallback)
            {
                ExternalValidationCallback evc = (ExternalValidationCallback) cb;
                // Add your password validation, unless bob works for you
                evc.setValidated("bob".equals(evc.getUsername() && "password".equals()evc.getPassword());
            }
            else if (cb instanceof AuthorizeCallback)
            {
                AuthorizeCallback ac = (AuthorizeCallback) cb;
                // Add your test to see if client is authorized to use requested
                ac.setAuthorized(true);
            }
            else
            {
                throw new UnsupportedCallbackException(cb, "Unrecognized Callback");
            }
    }
};

SaslServer ss = Sasl.createSaslServer("PLAIN", "myProtocol", serverName, null, myHandler);
ss.evaluateResponse(clientPacket);

Wednesday, February 3, 2010

Pivot Tree List labels and lazy loading

Here's how to lazy load a tree structure in pivot and supply nice(er) labels for all nodes. 1. Create a LabeledList that extends List
public class LabeledList extends ArrayList
{

private V label;

public LabeledList(V label)
{
    this.label = label;
}

public V getLabel()
{
    return label;
}

@Override
public boolean equals(Object o)
{
    return (o instanceof LabeledList && ((LabeledList) o).label.equals(label)
           && super.equals(o));
}

@Override
public int hashCode()
{
    int hash = 5;
    hash = 29 * hash + (this.label != null ? this.label.hashCode() : 0)
            + super.hashCode();
    return hash;
}
}
2. Create a simple renderer to display your new list
public class NodeTreeViewRenderer extends TreeViewNodeRenderer
{

 @Override
 public void render(Object o, Path path, int i, TreeView tv, boolean bln,
         boolean bln1, NodeCheckState ncs, boolean bln2, boolean bln3)
 {
     if (o != null && o instanceof LabeledList)
     {
         super.render(((LabeledList) o).getLabel(), path, i, tv,
                 bln, bln1, ncs, bln2, bln3);
     } else
     {
         super.render(o, path, i, tv, bln, bln1, ncs, bln2, bln3);
     }
 }
}
3. Put it together.
@WTKX TreeView myTree;
...
...
public void startup(Display display,
          Map map) throws Exception
  {
      ....
      List rootList = new arrayList();
      rootList.add(new LabeledList(new Date()));
      myTree.setData(rootList);
  }
Now we have the problem of loading all the child nodes when the tree is expanded. This is handled by creating a TreeViewBranchListener and expanding LabeledList a little to track if it's been loaded or not. In LabeledList, add the following:
    private boolean loaded = false;

  public boolean isLoaded()
  {
      return loaded;
  }

  public void setLoaded(boolean loaded)
  {
      this.loaded = loaded;
  }
Now create the Listener
public class LazyLoadBranchListener implements TreeViewBranchListener
{
  public void branchExpanded(TreeView tv, Path path)
  {

      List currList = tv.getTreeData();

      for (Integer i : path)
      {
          currList = (List) currList.get(i);
      }

      LabeledList lList = (LabeledList) currList;

     if (!lList.isLoaded())
      {
          // load data for into lList. Add additional LabeledLists for directories or
          // straight objects for end nodes.
          // For long running loads, do the load in another thread
          lList.setLoaded(true);
      }
  }

  public void branchCollapsed(TreeView tv, Path path)
  {}
}
In your application class, add the new listener
    myTree.getTreeViewBranchListeners().add(new LazyLoadBranchListener());